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ARMY TRAINING TECHZl.. 



GY TRANSFER: A SYSTEMS MODEL 



BRIEF 



Requirement : 

To improve individual and unit proficiency of Army personnel by devel- 
oping a systems model for effective transfer of training technology from re- 
searchers to Anr.y users. 



Procedure : 

Technology -transfer of Army training products was defined through analy- 
ses of military and nonmilitary literature and data. The background and his- 
tory leading up to the current interest of the Army in training technology 
transfer were presented in relation to pertinent data and critical events. 

A systems model of training technology transfer from both military and 
nonmilitary soirrces was described, along with the specific stages within 
each activity of the model. The presence or absence of relevant stages in 
the Army human resources research, development, test, and evaluation (RDT&E) 
program relative to the systems model was emphasized. Discussion and sug- 
gestions to improve Army training technology transfer followed. 

Findings : 

Development of the systems model revealed that several issues need to 
be addressed in Army training technology transfer. These issues are cate- 
gorized under general, specific, and recurrent topics. Under the general 
category, definitional, modeling, and transfer issues need attention. Under 
specifics, the issues that appear to need work include an assessment of needs; 
RDT&E funding; information dissemination; institutionalization; and predic- 
tion, monitoring, evaluation, and feedback. Recurrent issues needing atten- 
tion are sponsorship and a self -renewal capability. 

Utilization of Findings: 

This paper presents a conceptual view of the current situation of Army 
training technology transfer. Application of this systems model may allow 
Army decisionmakers to set priorities for addressing specific activities or 
issues related to the utilization of training products. 
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ARMY TRAINING TECHNOLOGY TRANSFER: A SYSTEMS MODEL 



OVERVIEW 



Operational Problem 



Parallel with the demand for new, costly, and complex Army materiel sys- 
tems is the requirement for insuring parity in the level of training neces- 
sary to operate 'and maintain these systems. This need for improved Army 
training has led to the research, development, test, and evaluation (RDT&E) 
of new training products. However, informal comrr.^nts and anecdotal evidence 
(Appendix A) suggest that significant numbers of recently fielded training 
products have not been integrated sufficiently nor used effectively to im- 
prove individual and unit proficiency within the Army's materiel systems 
(Drucker, 1977; Shields, 1976). One aspect of tko problem concerns the 
manner in which new training technology is transferred from researchers and 
developers to users. The emphasis on training technology transfer is based 
on the assumption that the strategies and procedures used in introducing new 
training products in the field determine their acceptance and use. 



Research Objective 



The objective of a successful training technology transfer program in 
the Army is the improvement of individual and unit proficiency in materiel 
systems. One means of achieving this objective is the provision of discus- 
sion and suggestions for improving training technology transfer. 



The Department of Defense (DOD) Technology Transfer Consortium was 
established in July 1971 as a result of congressional scrutiny of the extent 
of spinoff benefits from RDT&E, similar to benefits from the NASA space pro- 
gram (NASA, 1973; Tempest & Van Rooy , 1975). The consortium's purpose was 
to transfer military R&D results to solutions of civilian problems (between- 
agency transfer) rather than to concentrate solely-on military problems 
(within-agency transfer) . 

This paper focuses on the transfer of training technology from the Army 
human resources (KR) RDT&E community to the Army field user; consequently, 
the transfer of training technology from the Army to agencies outside the 
Army or DOD is not discussed. 1 Emphasis is on the role of the U.S. Army 
Research Institute for the Behavioral and Social Sciences (ARI) in providing 
accepted and useful HR RDT&E products to the user, as well as on a systems 



1 For discussicn of "between- versus within -agency transfer," see Jolly & 
Creighton, 1975; Millburn, 1979; Montanarelli , Jolly, & Creighton , 1977; 
Office of Technology Assessment, 1978; and Teich & Whartnaby, 1978. 



Scope 
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model describing nonsystem, advanced RDT&E with reference to basic and ex- 
ploratory development when relevant. 



Approach 

To present information about technology transfer from which Army de- 
cisionmakers can formulate a training technology transfer program, the 
following format is used: 

1. Background history of critical events leading up to the current 

interest of the Army in training technology transfer; « 

2. Description of a systems model of training technology transfer, 

from both military and nonmilitary sources; y 

3. Description of specific stages within each activity of the systems 
model, emphasizing the presence or absence of relevant stages in 
the current Army HR RDT&E program relative to the systems model; 
•and 

4. Discussion and suggestions to improve Army training technology 
transfer. 

Definitions 

In general, technology transfer can be defined as a process by which 
existing research knowledge is transferred operationally into useful proc- 
esses, products, or programs that fulfill actual or potential public or 
private needs (Federal Council for Science and Technology, 1975) , Key words 
in the literature used to describe technology transfer are change process, 
diffusion of innovations, innovation dissemination/transfer, knowledge 
sharing/transfer, product utilization, research dissemination/utilization, 
and technology assessment/sharing/utilization. 

The user is an individual and/or agency who (a) expresses or implies 
a need/requitement for a training product or (b) must approve, or transmit 
information of, the training product prior to its acceptance and use in the 
field. The researcher is an individual and/or agency involved in the RDT&E 
of a training product in response to an expressed or i.mplied user need. The 
training product is a research-derived set of procedures and/or devices de- 
signed to improve individual or unit proficiency in specified materiel sys- 
tems. These three terms are described more fully in Appendix B. For the 
purposes of this paper, Army training technology transfer is defined as a 
process by which the HR RDTf.ZT community produces an accepted and useful 
training product that fulfills the expressed or implicit requirements of 
the Army user; the main point is that successful training technology trans- 
fer results in a research product designed Uo produce a change. The research 
community may view this change in the way the user applies the training 
product; the user may view this change in terms of produced results within 
the user's operational environment (Table 1). 
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Table 1 



Differences in Researcher and User Meanings of 
"Use of Research Product" a 



Researchers 



Operational use of product 

Change-mission performance 

Incorporation of product 

Application of knowledge 

Adoption of resulcs 

Minimum translation 

Options "for development solution 



Users 



Improved effectiveness 
Cost-effective improvements 
Usable information 
Benefit to decisionmaking 
User requirements provided 
Saving of money 

Application of operational evidence 



Based on a survey of 67 Amy, Navy f Air Force, and contractor researchers 
and product users (Drucker, 1977) . 

Since the most visible result of training technology transfer is the 
training product, interest has been focused on the dissemination and eventual 
use of the training product. However, this report shows that the training 
technology transfer process involves both formulating the training product 
and introducing it to the Army user. This system concept document, in a 
"top-down" framework, supports the establishment of an Army training tech- 
nology transfer program and presents a conceptual view of the current situa- 
tion of training technology transfer in the Army and other areas. The "big 
picture" of the training technology transfer process will allow Army decision- 
makers to determine which specific activities in the program may need 
improvement. 



BACKGROUND 

POD Involvement 

During the past two decades, research and development (R&D) of major 
materiel systems and associated training products have become increasingly 
specialized. This specialization has brought a greater emphasis on the 
interdependence between the R&D community and the user community (Clarkin, 
1978). For example, in 1966, President Johnson stated that 

Presidents . . . need to show more interest in what the specific 
results of research are in their lifetime, and in their administra- 
tion. A great deal of basic research has been done . . . but I 
think the time has come to zero in on the targets — by trying to get 
our knowledge fully applied. ... We must make sure that no life- 
saving discovery is locked up in the laboratory. (Comroe & Dripps , 
1975) 
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This statement forecasted the Federal attitude toward a greater emphasis 
on the usefulness and applicability of research products. 

The Department of Defense conducted a study, "Proje< t Hindsight," in 
1966 to track the use of specific military R&D products (Comroe & Dripps , 
1975) . The study analyzed retrospectively the development process of 20 
weapon systems (e.g., Polaris and Minuteman missiles, M102 Howitzer). Some 
of the study's conclusions were that (a) contributions of basic, scientific 
research were minimal; (b) scientists contributed most effectively when 
their effort was mission-oriented; and (c) the time lag between initial 
discovery and final application was shortest when the scientist worked in 
areas targeted by the sponsor. Although this study focused on hardware 
products (e.g., materiel, weapon systems) rather than on software products 
(e.g., training programs), it indicated initial military involvement in 
evaluating the usefulness and relevancy of its research products. During 
this time, new terms were popularized in the general R&D community to re- 
flect this trend: targeted research, mission-oriented research, program- 
matic research, commission-initiated research, contract-support research, 
and payoff research. Congress and the Office of Management and Budget re- 
acted accordingly with an increase in NIH-supported contract research and 
commission-initiated research (Comroe & Dripps, 1975). Thus, during the 
1960's, the trend was set for justifying applied research in terms of user 
needs, stated in terms of goals, targets, etc. 

In the late 1960 , s and early 1970 , s, Congress showed an increased in- 
terest in the relevancy of R&D products. For example, the Mansfield Amend- 
ment to an early 1970 appropriations bill stated that all defense R&D must 
have a direct and apparent relationship to the DOD mission. This relation- 
ship apparently emphasized the overall DOD mission, in the development of 
hardware, weapon systems or subsystems, or in the maintenance of combat 
readiness through better training, better personnel management, or better 
manpower management (Taylor, 1978). In response to this amendment, each 
of the military services reviewed their R&D programs based on categories 
which had (a) both a direct and apparent relationship, (b) an apparent re- 
lationship,, or (c) neither a direct nor an apparent relationship to the 
overall DOD mission. This intraservice review resulted in a restructuring 
of the military R&D program from the standpoint of both relevancy and em- 
phasis (Taylor, 1978) . 

In 1971, the Ginhesburg Management Report recommended an increase in 
R&D expenditures in the behavioral and social sciences and improvement of 
the management in these areas. Some of the findings of the report were 

(a) a lack of communication between policy decisionmakers and researchers; 

(b) too much control of research by the hardware community; (c) too much 
research originating from the bottom up, rather than from the top down; 

(d) lack of involvement and control by those responsible for the program; 

(e) lack of responsiveness to the needs of the client; and (f) too much 
concentration on research in the hands of particular groups (here, the 
Ginnesburg report focused on psychologists and the lack of involvement of 
a wide variety of disciplines (Rostker - 1978)). 

The next major historical point was the House Appropriations Committee 1 
review and report on the 1976 fiscal budget request (Taylor, 1978) . The 
committee recommended a 50% reduction (from $40 million to $20 million) 
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within human resources R&D due to the lack of demonstrated utility of re- 
search products in this area. However, the Senate restored half of the 
$20 million reduction, and the House acceded to the Senate's position (Tay- 
lor, 1978). Consequently, the budget for human resources R&D was kept in- 
tact, but Congress caught thi attention of the R&D community. Since t.iat 
t-me there has been an effort within the R&D management community and within 
certain segments of the user community to clarify the use of human resources 
R&D (Taylor, 1978) . 

In November 1975, following the final uthorization of the FY 1976 bud- 
get, the Chairman of the House Appropriations Committee requested that the 
General Accounting Office (GAO) conduct a detailed review of the military 
human resources R&D program so that a more thorough analysis of the Ff 137/ 
defense budget request could be completed (Reusse, 1976; Taylor, 1978). On 
April 22, 1977, the GAO report entitled "Human Resources Research and De- 
velopment Results Can Be Better Managed" was published. 

The GAO conducted its evaluation over a period of about 18 months. They 
developed an audit trail of human resources research, using the technical 
report as a basic source to audit. They visited eight organizations within 
DOD and identified 374 reports that contained results which the research 
community deemed ready for immediate use. To complete the audit trail, the 
GAO contacted users both by survey and personal interviews to determine which 
R&D res- -'ts were used and which were not used. They found that (by their 
criteria) 56% of the results had been used and 38% had not been used, with 
6% considered for possible use. The GAO emphasis was on immediate use and 
did not consider results relating to the decisionmaking process (Taylor, 
1978) Some of the reasons thr.t the GAO report found for nonuse were that 
(a) users believed that results were for information only; (b) users had 
not seen the reports or were unaware of the reported results; and (c) the 
research community questioned the results, or they believed them to be un- 
usable (Taylor, 1978). 

As a result of this study, GAO submitted to DOD four recommendations 
intended to improve the management of research utilization: 

1 Develop criteria to identify R&D results with immediate (i.e., 
near-term) use potential. The GAO used three criteria for de- 
termining use potential of a given research project, i.e., re 
search intended to support ch^jige in education or training; 
equipment; or regulations, order, policy, doctrine, or manuals. 
One idea is ~o require that each published technical report in 
the human resources area provide a statement concerning the in- 
tended user or user community and the suggested use of any product 
or findings. 

2. Improve communication between researchers and users. Some ideas 
are periodic utilization reports; user involvement through tech- 
nology process; collocation of researchers with users; conferences; 
and linkage agents serving as intermediaries between R&D and user 
communities . 

3. Establish a monitoring and feedback system for tracking utilization. 
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4. Develop a management, mechanism for resolution of issues between 
the researcher and the user (Reusse, 1976; Taylor, 1978) r 

Similar recommendations were put foi/Lh by the 1976 Defense Science 
Board Task Force on Training Technology (Alluisi, 1977) and the Laboratory 
Utilization Report (White & Taylor, 1976), with the additional recommenda- 
tions of the need for (a) cost-benefit and performance-effectiveness analy- 
sis relating to research utilization; (b) resolution of conflict of interest 
between researcher and user; and (c) emphasis on the whole R&D process, from 
requirements to utilization (Taylor, 1978) . 

A major response by the DOD human resources RDT&E communities to the 
growing Federal concern over the use of research products was the convening 
of the National Symposium of the Military Services on Utilization of People- 
Related RDT&E in June 1977 (Sands & Glaser, 1978). The purpose of the sym- 
posium was to assess the problems of human resources RDT&E, principally 
within the military establishment, and to formulate recommendations for im- 
proving utilization strategies of RDT&E. Recommendations from chis sym- 
posium emphasized improvement and/or increased activity in the following 
areas: (a) use of research products, (b) interaction between researchers 
and user communities, (c) accountability and responsibility affecting the 
use of research products, (d) relevancy to R&D requirements, (e) implementa- 
tion planning, and (f) reporting and dissemination of research findings. 

Army Involvement 

In December 1972, the Military Requirements and Product Utilization Of- 
fice was formed within ARI to assist in cooperative efforts between the re- 
search technical areas and the sponsor-users of the results of ARI research. 
The staff consisted of . military officers who performed liaison functions. 
The purposes of the office were to help define needs and problems and to 
promote the adoption and effective use of the research products (Uhlaner, 
1977). \7his office was subsumed eventually under the Plans, Programs, and 
Operations Office in ARI. A major contribution of this office was to heighten 
those activities concerned with promoting the use of ARI research products, 
e.g., the Technical Advisory Service, the Tri-Service Utilization Service, 
the draft research utilization plan, and a task force to study the use of 
research products. 

Dusek (1974) traced the history of technology transfer in Army human 
resources R&D. Although Dusek 1 s review emphasized between-agency transfer, 
this historical survey is important because it provides an accounting of 
significant contributions developed by the Army human resource RDT&E 
community. 

Two major Army regulations defining doctrinal and procedural aspects 
of personnel performance and training RDT&E were AR 70-1 (1975) and AR 70-8 
(1976) . These regulations defined the extent of ARI responsibilities in 
performing RDT&E, the program formulation and approval process, the RDT&E 
categories, and the implementation of RDT&E fundings. Major research funding 
categories were delineated within ARI's purview, namely basic (6.1), explora- 
tory (6.2), and nonsystem advanced development (6.3A) . This last category 
focuses primarily on non-material-oriented technology (e.g., for training 
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products) which has potential application to a variety of systems rather 
than to one specific system. 

In May 1976, ARI research scientist s participated in the Potomac Chap- 
ter of the Human Factors Society annual symposium on training technology 
transfer, which addressed research and policy problems in education and 
training. This meeting produced the explanation and tracing of training 
from the RDT&E phase through the process of translating the resultant tech- 
nology into implementation policy to the impact of policy in training. 

Drucker (1977) presented the results of a survey on military research 
product utilization to the Military Psychology Division of the American 
Psychological Association in September 1976. Major findings from this sur- 
vey were (a) that researchers and users have different meaningr, for the 
term "research product utilization," (b) a breakdown of different descnp- 
tions of users, (c) factors influencing the acceptance and use of a research 
product, and (d) discussion of evaluation criteria for product utilization. 
Shields (1976) discussed the process of technology transfer in relation to 
the Army's need for improved (and timely) training products. Her review 
presented deficiencies in existing models of technology transfer (e.g., 
implementation, timeliness), factors to be considered in an Army model of 
training technology transfer (e.g., incentive system, organizational struc- 
ture^, and current status and future trends within the Army. Shields re- 
view' provided the framework from which an Army model of training technology 
transfer could be developed. \ 

In summary, both DOD and Army involvement in training technology trans- 
fer has focused on the ways to insure the acceptance and usefulness of the 
training research product. Although a consensus had developed as to this 
goal, the particular means or sequence of activities necessary and sufficient 
to achieve the goal had not been delineated. Based on a literature review, 
this report describes important activities in promoting the acceptance and 
usefulness of a training product. Deficiency in any one of these activities 
may result in unsuccessful implementation of the product. 



SYSTEMS MODEL 

Based on a review of the literature on technology transfer and of Army 
regulations pertaining to HR RDT&E, four major sequential activities were 
considered to be sufficiently characteristic of the Army training technology 
transfer process (Figure 1) . The activities are defined and their associated 
key words presented in Table 2. References are categorized by the primary 
activity emphasized (Appendix C) . The model was developed in accord with 
the following considerations. 

1. The overall intent of the model was primarily prescriptive, i.e. , 
informing the reader what is done, as exemplified in the major 
activities and their stages. Subsequent discussion and suggestions 
were designed to be predictive , i.e., informing the reader of al- 
ternative methodologies that may improve certain activities. 

2 Primary focus was on describing the appropriate sequence of activi- 
ties involving the initial formulation of a proposed researchable 
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MONITORING, EVALUATION , AND FEEDBACK 



Analyze 
Requirements 



Research , Develop , 
Test, and Evaluate 
Solutions 



Disseminate 
Information 



:ut ; nalize 
;i inc i.ngs 



Researchable 
Question 



Training 
Product 



User 
Acceptance 



Figure 1. Army training technology transfer model. 
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Table 2 



Description and Key Words of Each Activity in the Model 



Activity 



Description 



Key Words 



Analyze requirements 



A systematic effort by the researcher and the user 
to determine the goals, objectives, or alternatives 
for the basis of a productive effort at applied re- 
search technical assistance or organizational im- 
provement (Hambrick, 1978). Requirements analysis 
may be viewed as a discrepancy analysis between 
"What is" (current condition; baseline activity 
level) and "What should be" (required or desired 
condition) . A problem may then be defined as a 
documented discrepancy selected for resolution 
(Kaufman, 1972), Techniques to assess requirements 
must yield information that attempts to represent 
the two polar conditions of "What is" and "What 
should be." The result of a researoher-user re- 
quirements analysis is a researchable question, 
directed by the user's needs, and adapted techni- 
cally by the researcher's experience. 



Baseline 

Definition phasf; 

Discrepancy analysis 

Forecasting 

Front-end analysis 

Needs analysis/assessment/ 
identification 

Organizational diagnosis 

Planning stage 

Problem analysis/defini- 
tion/diagnosis/ 
identification 

Program analysis/formula- 
tion 

Project formulation 
Social & bek'.'ioral 
indicators 



Research, develop, 
test, and evaluate 
solutions 
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A systematic effort to establish a scientific knowl- 
edge base (6.1) for potential solutions to specific 
military problem areas (6.2) and to apply this knowl- 
edge in solving a researchable question directed, in 
part, by a military sponsor's need and/or directive 
(6.2A). The procedures describing Army HR RDT&E are 
the most documented (by regulation and doctrine) 
relative to the other activities in the model. The 
end result of this activity is a training product 
that satisfies a sponsor's requirement. 



Applied research a 
Assessment of casualty 
Basic research' 3 
Design, development 
Factor identification/ 

manipulation 
Innovation/ invention 

stage 

Technology application/ 
utilization 
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Activity 



Table 2 (Continued) 



Description 



Key Words 



Research, develop, 
test, and evaluate 
solutions (Continued) 



Disseminate information The dispensing of information about RDT&E products 

to users at various distances from the points of 
origin of the RSD product (Shields, 1976). An ARI 
scientist can relay information about a particular 
training product to the military sponsor (s) who 
originally requested a need for the product, de- 
scribing the product designed in response to the 
user's need, demonstrating its operation, provid- 
ing assistance in training "front-line" users to 
operate the product, and turning over the whole 
package to the users for their own purposes . The 
ARI scientist can also inform other units, commands, 
agencies, etc., about this product, thereby broadly 
' diffusing the findings to potential users remote 
from the initial application of the training prod- 
uct. Individuals who promote the acceptance of the 
training product into their/other organization (s) 
are called change agents or linkage agents , and the 
process whereby disseminated findings are convinc- 
ingly demonstrated to, and by, the change agents 
(and other users) is called linkag e or change 
agentry . Indicators of the dissemination of infor- 
mation to the user are observed in (a) professional 
publications , technical reports, briefings, and 
meetings with the sponsor/user; and (b) use of the 
Army Research and Development Information System 



Prototype model/breadboard 

mock up 
Variable relationships 
Field testing and 

evaluation 
Validation and feedback 

Change agentry 

Communication 

Confirmation 

Decision 

Demonstration 

Diffusion 

Exchange/feedback 

Flow of information 

Knowledge flow 

Linkage 

Reception/rejection 
Retrieval/memory bank 
Transmission - . 



Table 2 (Continued) 



Activity 



Df scription 



Key Words 



Disseminate information 
(Continued) 



Institutionalize the 
findings 



(ARDIS) via its two subsystems: The Management In- 
formation System (ARDIS-MIS; provides management 
type information to DCSRDA and information and guid- 
ance to ODCSRDA and other Army R&D managers), and 
the Scientific and Technical Information Program 
(SSTI; supported by the DDC data bank). The end re- 
suit of this activity is the user's acceptance of 
the training product, 

Following user acceptance, the time period during 
which the training product is incorporated and used 
effectively by the Army user, Ultimately, the 
training product becomes a stable and regular part 
of Army organizational procedures and user behavior. 



Adaption 

Adoption 

Application 

Assimilation 

Diffusion 

Distribution 

Implementation 

Policy 

Routinization 
Utilization 



Perform monitoring, 
evaluation, and 
feedback 



A systematic effort to monitor and evaluate the 
technology transfer process of a training product, 
and to provide feedback to the researcher and user 
concerning changes and new requirements in the 
formulation and introduction of current and sub- 
sequent training technology. 



Assessing the level of 

product use 
Evaluation study 
Implementation study 
Predictive model of tech- 
nology transfer 
Project monitoring 
Program evaluation 



Also known as commission-initiated research, contract-supported research, directed-research, mission- 
oriented research, payoff research, targeted research, research in the service of man, technology. 



Also known as contracted/grant research, fundamental research, nontargeted research, undirected research, 
science research. 



personnel management and training system for soldiers; (d) techniques in 
training literature; and (e) a network for exchanging ideas and requirements 
between operational units and the training base responsible for providing 
instructional materials and resources (Roberts, Daubek, & Johnston, 1977). 

Four years later, a symposium was held at Fort Eustis, Va. , to identify 
problems in instructional services provided by Army schools (Roberts et al . , 
1977) . The 1975 Instructional Technology Symposium report specified solution 
strategies that were applied Army-wide to create a new agency, the Training 
Development Institute (TDI) of the U.S. Army Training and Doctrine Command 
(TRADOC) . The solution strategies specified by the 1975 report essentially 
consisted of two programs: (a) expert instructional technology consultation 
and services to Army schools, and (b) comprehensive staff and faculty train- 
ing programs. These programs have had an impact on other Army training de- 
velopment programs, such as self-paced instruction and the Integrated Techni- 
cal Documentation and Training (ITDT) program; the latter is a program to 
change technical publications from merely reference manuals into training 
resources for soldiers (sea also Army Research, Development, a>nd Acquisition 
Magazine, 1978) . 



Research, Develop, Test, and Evaluate Solutions 



The stages of the RDT&E solutions activity are detailed in Army Regula- 
tions 70-1 and 70-8. Figure 3 presents a general outline of the research 
funding efforts' impact on non-system-advanced development (6.3A). A more 
elaborate breakdown of the training RDT&E effort was described by the Train- 
ing Development Working Group of ARI (1977) . This group applied the frame- 
work of the Life Cycle System Management Model (LCSMM) (DA Pamphlet No. 11-25, 
1975) of Army materiel system RDT&E to the training RDT&E effort. As Fig- 
ure 4 shows, this model primarily emphasizes what the present paper subsumes 
under the RDT&E solutions activity. The major contribution of the LCSMM 
training model appears to be as procedural markers or guidelines in conjunc- 
tion with the policy statements in Army Regulations 70-1 and 70-8. The 
ma s ' a or theme throughout the RDT&E solution activity is that a well-defined 
rosearchable question should be systematically, investigated and developed 
into an evaluated training product, guided by researcher-user involvement. 



Disseminate Information 

Dissemination of information about a training product may proceed along 
two main paths. The first path is one in which communication is directed to 
persons other than the user who initially requested the RDT&E of the training 
product. The purposes here are primarily to inform and secondarily to per- 
suade. Such activities can be found currently in the Army Research and De- 
velopment Information System (ARDIS) , which includes management information 
for DCSRDA (MIS) , and scientific and technical information for scientists 
and other interested individuals (S&TI) . Also, ARI scientists publish tech- 
nical reports, provide technical advisory service, and administer the U.S. 
Army Human Factors Research and Development Conference in this regard. The 
second path is concerned with both informing and persuading the user to ac- 
cept the training product. Here, briefings and informal dialogs are just as 
relevant as organizational sponsorship and budgetary considerations. The 
stages for information dissemination are presented in Figure 5. 
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"Top-Down" 




"User-generated" 
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DOB Programs 
ARI Directives 



new 
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6.3A 



Training Product 
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i — 




6.1 Themes 
ILIR 





"Bottom-Up" 



Figure 3. Activities impacting on RDT&E solutions in 6.3A research. 
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A significant amount of research has been conducted to develop models 
predictive of user acceptance through the process of dissemination of in- 
formation about the trairing product. Thus, research has focused on vari- 
ables influential in this process. As noted in Appendix B, the level of 
involvement of the user to which communication is directed is an important 
variable. Moreover, consideration of relevant factors influencing this 
process have been cataloged in the literature (see references in overviews 
and models section for more detail). For example, Table 3 presents a repre- 
sentative list of relevant factors underlying the success of the dissemina- 
tion of information for eventual user acceptance of the training product 
(Shields, 1976) . These factors could be applied as a checklist indicating 
either the presence or absence of these factors during training technology 
transfer, or rated along a dimension of visibility, such as completely ab- 
sent, somewhat visible, completely present. 

Another use of these factors has been more quantitative and predictive 
in purpose. This approach attempts to translate qualitative, descriptive 
factors deemed important for technology transfer into quantitative, numeri- 
cal values which can be incorporated in a mathematical equation, whose re- 
sult would provide an index representative of the predicted success/failure 
of the acceptance or use of a training product by the user. An example of 
this approach is provided by the Linker Model of Jolly and Creighton (1977) 
presented in Table 4. The Linker concept is used in this model as a term 
for effective technology transfer. The linking mechanism necessary to 
achieve effective technology transfer is described by identifying the fac- 
tors that contribute to predicting the movement of technology from the 
source of knowledge (researcher) to the use of knowledge (user /receiver) . 
The nine factors influential in the dissemination process are divided into 
two classes, formal and informal factors. Formal factors consist of pro- 
cedures for dissemination of storage, indexing, and retrieval of knowledge. 
These procedures may consist of different types of informative reports de- 
signed for specific levels of uses, data banks, and/or user feedback/input 
documentation which influences the RDT&E of the training product. Formal 
factors are generally visible and easy to catalog and measure. Informal 
factors involve interpersonal communications and contacts; personal beliefs 
and feelings about a knowledge source; and perceptions about one's organiza- 
tion, supervisors, and peers. In forma 3. factors are essentially behavioral 
in nature, measured by subjective reports that attempt to scale attitudes 
and response evaluations of training technology transfer. These factors are 
listed and described below. 



Formal Factors 

1. Method of Information Documentation : Refers to how the technical 
information is recorded; rated by the format used, organization of 
material, complexity of the language, ease of indexing and retrieval 
(e.g., ARDIS, MIS systems, executive summary style). 

2. The Distribution System : Refers to .the physical channel through 
which information flows (e.g. , formal distribution lists, publica- 
tions in journals, symposia presentations and conferences, informal 
meetings) . 
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Table 3 



Factors Within Each of the Major Categories of Variables Which 
May Influence Technology Transfer in the Arm^ 



Innovation Factors 



Organizational Factors 



Individual Factors 



Type of innovation 
(hard vs. soft) 

Source of innovation 

Match with operating 
environment, capa- 
bilities of human 
operators and oLher 
elements of the 
system 

Reliability 

Complexity 

Effectiveness 

Documentation 

Ease of use 

Capital, personnel, 
and time 
requirements 

Visibility 

Adaptability 



Size 

Formal structure 
Informal structure 
Attitude toward 

authority 
Goals , values , customs 
Cohesiveness 
Communication patterns 
Communication with 

change agent 
Organizational 

interdependency 
Resources 



Education/training 
Intellectual level 
Status in group 
Alternatives 
Attitude toward 

change 
Attitude toward 

change agent 
Attitude toward 

authority 
Self-esteem 
Satisfaction with 

intrinsic 

motivators 
Cosmopolitanism 
Exposure to outside 

attitudes 
Attitude toward 

present equipment/ 

system 
Attitude toward 

innovation 
Resources 



Implementation Fac tors 



External Factors — Milieu 



Presence of a change agent 
Communication channels used 
Vigor of the communications effort 
Form of communications (content) 
Characteristics of the change agent 
The supplier 

Completeness of innovation package 

Implementation delays 

Prior need 

Command support 

User participation 

User training 



General social conditions in 

society 
General political conditions 
Critical events, crises, and 

re volutions 
Acting on the adopting 

organization 



a Shields, 1976. 
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Table 4 



Predictive Model of Technology Transfer (The Linker Modex 
of Jolly and Creighton, 1977) 






Method of Information 


DOCU 


"l"l 




Documentation 








6 2 C 2 




The Distribution System 


DIST 












9 3 C 3 




Formal Organization of the 


ORGA 




User 








°4 C 4 




Selection Process for Projects 


PRO J 




(Users ' Contribution) 








6 5 C 5 




Capacity of the Receiver 


CAPA 












G 6 C 6 




Informal Linkers in the 


LINK 




Receiving Organization 












Credibility as Viewed by 


CRED 


G 7 C 7 




the Receiver 












| Perceived Reward to the 


REWA 


8 8 




Receiver 








e c„ 




Willingness To Be Helped 


WILL 


9 9 










The model may be expressed in equation form such that: 



L . 
l 



E9 1 c l + e 2 c 2 + + + G.c k 



Where 



Note : 



Lj[ = Linker index for an organization i 

6 j = A measure of factor utilization, Oj range 0 1 

C k = A measure of the factor contribution, EC^ = 1 

The linking mechanism necessary to achieve effective technology trans- 
fer is described by identifying the factors that contribute to movement 
of technology from the source of knowledge (supplier) to the utiliza- 
tion of knowledge (user/receiver) . 
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3. Formal Organization of the User : Refers to the user's perception 
of his/her formal organization's influence on the use of technical 
information; consider infrastructure elements, such as the power 
structure, the nature of the business, the management style, the 
resources available, management's attitudes, the amount of bureau- 
cratic procedures, and the stability of the organization (e.g., 
dependent on levels of users) . 

4. Selection Process for Projects (User's Contribution) : Refers to 
the user's input to the selection of the R&D project (e.g., feed- 
back, HRNs) . 



5. Capacity of the Receiver (User) : Refers to the individual charac- 
teristics associated with the extent of use of a training product 
(e.g., traits, attitudes of a person, such as leadership qualities, 
educational experience, age, social status, rank, etc.). 

6. Information Linkers in the Receiving Organization : The person or 
persons who promote technology transfer process (e.g., see Appendix 
B, such as sponsor's representative, commandant, front-line user, 
congressional staffer) . 

7. Credibility as Viewed by the Receiver : Refers to the user's per- 
ception (assessment) of the reliability and accuracy of the informa- 
tion (associated with the training product) . Credibility is a 
function of the perceived reliability and accuracy of both the 
source and the channel through which the information flows. The 
extent of use and the rate of adopting research output correlates 
with the credibility of the available technology. 

8. Perceived Reward to the User : Refers to the perceived and actual 
recognition of using an innovative product in the organization cf 
which the user is a member; can be divided into two broad categories: 

(a) intrinsic — opportunity to use skills, to gain new knowledge, to 
deal with challenging problems, and to have freedom to follow up 
one's own ideas (akin to self-growth in competence tendency) ; 

(b) extrinsic — monetary reward, increased administrative authority, 
association with top executives, etc. (It is suggested later that 
technology transfer in the Army has relied more on the intrinsic 
reward system than on the extrinsic reward system.) 

9. Willingness To Be Helped : Awareness, or familiarization, of an in- 
novative product is not sufficient to insure acceptance and use of 
the product (e.g., Freda & Shields, 1979). There must be a self- 
interest (internal motivation) .to improve one's operational setting 
via the use of new training procedures. 

The Linker Model's main contribution is the recognition of both procedural 
(formal) and behavioral (informal) factors influencing successful dissemination 
of information about a training product. Jolly (1975) measured these factors 



Informal Factors 
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by studying the responses selected organizations made to questions designed 
to describe each factor- Organizations were * differ statistically 

on several of these factors — organizations defined as having employed ef- 
fective technology transfer beforehand differed in these responses, relative 
to those organizations that were not effective. The significance of this 
research is that it is a first, but essential, step in predicting the suc- 
cess of technology transfer. One issue that needs to be resolved with this 
prediction technique is the development of a reference standard (baseline) 
against which to judge technology transfer based on the aforementioned fac- 
tors in the model. 



Institutionalize the Findings 



After user acceptance, it is important to note the differences between 
the initial utilization and subsequent implementation of the training product 
(Drucker, 1977) . Utilization refers to cooperative efforts between the re- 
searches and user to incorporate the training product within the operational 
setting. This stage can involve 6.1 and 6.2 efforts to support 6.3A success. 
Implementation involves further use, operational adaptation, and expansion of 
the use of the training product. This activity is generally performed by 
the users above, following 6 . 3A efforts, and it is not generally a formal 
function of human re .ources RDT&E. 

The current IjDT&E funding categories present a "Catch-22 ,v situation for 
the HR RDT&E community. On the one hand, congressional scrutiny of military 
training budgets forces the research community to justify its existence by 
demonstrating effective use of its training products. On a research level, 
this goal is translated into developing objective measures of users 1 benefits. 
On the other hand, the fiscal control lent to the HR RDT&E community officially 
stops at the 6.3A funding category (nonsystem advanced development) , and fur- 
ther control of the training product is taken over or dispersed by other agen- 
cies from, the 6.4 (engineering development) to 6.7 (operational system devel- 
opment) funding categories. This procedure carries over from weapon-systems 
development practice, in whici) software (e.g., training) development is made 
equivalent to the hardware (e.g., materiel system) development process. [One 
aspect of this analogy is the difference in visibility, for justification 
purposes, between a software (nontangible) product and a hardware (tangible) 
product.] Therefore, if the HR RDT&E community is to be held accountable 
for .the acceptance and effective use of their training products, it would 
appear reasonable to allocate to that community a significant amount of 
budgetary control over the dissemination, utilization, and implementation 
of their training products. At present, Congress is asking the HR RDT&E 
community to be responsible for the operational effectiveness of their train- 
ing products, over which that community has no formal budgetary control. 

An example of institutionalization (Drucker, 1977) is a personnel test 
that is critically validated (utilization) and, after the user receives it, 
is subsequently standardized on a larger scale (implementation) . Another 
example is a prototype training course developed for a particular training 
manager (utilization) which is then used by the training manager for wide- 
spread application in a number of training locations (implementation) . The 
final stage is policy, which is not a formal part of the HR RDT&E effort. 
During the policy stage, use of the training product has become standard 
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practice , defined and supported by Army regulations. When this happens , 
use of the training product has become ,f rcutinized r (Yin, Quick, Bateman, 
& Marks, 1978). The stciges of the institutionalization of findings are 
presented in Figure 6. 



Utilize the 
Product 



Implement the 
Product 



■3 



Policy Directs 
the Use of 
the Product 



Figure 6. Institutionalize the findings. 

Yin et al. (1978) have attempted to quantify and measure the stages of 
the institutionalization of an innovative product. They contend that an 
emphasis o;i this activity is lacking in the current RDT&E approach to tech- 
nology transfer. To facilitate the analysis of the institutionalization 
process, the process of eventual routinization is viewed as a composite of 
both passages and cycles. A passage occurs when a formal transition from 
one organizational state to another has been accomplished. For example, 
establishing new procedures for instructional development constitutes a pas- 
sage. A passage reflects increased organizational support for an innovative 
product, generally occurs only once, and is indicative of its having become 
a more integral part of the organization. In general, the more passages that 
have been achieved, the more routinized a product may be regarded. A cycle, 
on the other hand, is an organizational event that occurs repeatedly during 
the lifetime of an organization. In general, the more cycles that an inno- 
vation survives ? the more routinized it is. Thus, "cycle 1 ' applies to repeated 
events that occur as part of an organization's operations and that may affect 
an innovative product. 

Table 5 presents Yin's breakdown of routinization in terms of resources 
and operations specific to passages and cycles. Observation or use of these 
resources/operation occurs during particular stages of the routinization 
process. Similar to the stages of the institutionalization process are the 
stages of the "life histories' 1 process underlying routinization. These 
stages are improvisation (utilization) , expansion (implementation) , and 
disappearance (policy) . The improvisation stage is the initial period, fol- 
* lowing user acceptance, during which the product is used,, and no significant 

passages or cycles need occur during this stage. The main goal here is to 
use the product at some meaningful level/frequency for some specified period 
of time. The expansion stage is characterized by the growing use of the 
product and the achievement of several passages and cycles. Finally, the 
disappearance stage involves the completion of the remaining passages and 
cycles. During this stage, the product continues to be used but eventually 
loses its recognition as an innovative product — i.e., it achieves- the status 
of standard agency practice (policy) . 

Table 6 organizes the occurrence of passages and cycles, along with 
their respective resources/operations, into specific stages of the life 
history of an innovative product. Table 7 presents 10 factors associated 
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Table 5 

Organizational Passages and Cycles Related to Routinization 



Type of resource or operation 


Passages 


Cycles 


1. Budget 


Innovation supports changes from 
soft to hard money (2) a 


Survives annual budget cycles 


2. Personnel 
Jobs 


Functions become part of job de- 
scriptions or prerequisites (5) 




Incumbent turnover 




Survives introduction of new 
personnel (9) 

sonnel (8) 


3. Training 
Pre-practice 


Skills become part of profes- 
sional standards, professional 
school curriculum (7) 




In-service 


mm 


Skills taught during many train' 
ing cycles 


4. Organizational Governance 


Establishment of appropriate 
organizational status (3) 

Use of innovation becomes part 
of statute, regulation, manual, 
etc. (6) 


Attainment of widespread use 
(10) 


5. Supply and Maintenance 


Supply and maintenance provided 
by agency or on long-term 
(contract) basis (4) 


Survives equipment turnover (1) 


lumbers refer to 10 passages and cycles studied in actual life histories (Yin et al., 1978),. 



Table 6 

Summary of Passages and Cycles in an Innovation's Life History 



raboayc 




Number of case studies 


ay* ntini a 

or cycie 




achieving each 


uunujer 


Life history stage 


passage or cycle 




Improvisation stage 


• 




(no necessary passages or cycles) 






Expansion stage 




I 


Equipment turnover 


13 


I 


Transition to support by local funds 


18 


j 


Establishment of appropriate 






organizational status 


14 


A 

4 


Establishment of stable arrangement for 






supply and maintenance 


15 


5 


Establishment of personnel classifications 






or certification 


12 




Disappearance stage 




6 


Changes in organizational governance 


7 


7 


Internalization of training program 


6 


8 


Promotion of personnel acquainted with the 






innovation 


6 


9 


Turnover in icey personnel 


7 


10 


Attainment of widespread use 


10 


Source: Yinetal., 


1978. 
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Table 7 

Factors Associated with the Facilitation of the 
Routinization of an Innovative Product 



1. Core application in an agency's activities 

• Determine by observing if innovative product has displaced some 
significant function 

2. Minimal competition for resources among different applications 

3. Service payoffs; should include 

• Specific nature 

• "Perceived by many people 

• Consensus that payoff exists 

4. Prior need for innovative product 

5. Client support 

6. Community support 

7. Top administrative support within the agency 

8. Practitioner support within the agency 

9. Active innovator support 

10. No adversary group that specifically opposes the innovative product 

Source: Yin et al., 1978. 

Note: Factors 5, 6, 7 , and 8 can be used in relation to the different levels 
of us6rs presented in Appendix B. 
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with the facilitation of routinization of an innovative product (Yin et al., 
1978) . 



Table 8 displays the extent of influence these 10 factors had on routini 
zation in various locations using different innovations. The routinization 
process is graphically portrayed in Figure 7* 

The relevance of Yin's approach to Army training technology transfer 
resides in th# use of organizational indicators to define the extent that a 
training product is successfully incorporated within an operational setting. 
Within this framework, reliance upon indicators such as utilization reports 
(AR 70-8) provides only a partial picture of where the product stands in the 
way of incorporation. Other indicators could be modified and incorporated 
by Army decisionmakers into an "institutionalization checklist," which would 
provide a documented metric of product utilization. 

Monitoring, Evaluation, and Feedback 

This activity is ongoing throughout all the main activities of the model 
For example, during requirements analysis, emergent requirements from the 



field or policy makers can be evaluated and possibly incorporated into the 

formulation of ongoing' RDT&E efforts. During RDT&E, user-researcher dialog 
can result in review of training design respecif ication for possible incor- 
poration into ongoing R&D efforts. During dissemination and institutionali- 
zation, researcher and user feedback can provide information to guide the 
acceptance and effective use of a training product. Models predictive of 
technology transfer are also used to measure the extent of successful incor- 
poration of a training product during institutionalization. Methodologies 
employed in this activity consist of multivariate prediction models, on-site 
interviews, and monitoring and user feedback reports. A major point of em- 
phasis is that this activity should be continuous, even during the institu- 
tionalization process. Once a training product is defined as standard prac- 
tice (policy or routinization) , monitoring and evaluation of this practice 
can result in a new emergent requirement, which in turn initiates the train- 
ing technology transfer process. Notice that, as previously mentioned, the 
training technology transfer process is cyclical in nature, and only linear 
for explanatory purposes. 



The following topics present discussions of the aforementioned model, 
and suggestions to improve Army training technology transfer. 

Organization of Regulations and Procedures 

The procedures defining certain aspects of Army training technology 
transfer are in several documents. It is suggested that an ARI regulation 
could consolidate relevant sections of Army regulations in accordance with 
the model presented in this paper. A general outline is presented in Table 9 
to help in this endeavor. 
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Table 8 



Presence of 10 Factors Hypothesized to Facilitate Routinization (Case Studies) 



Ten factors hypothesized to facilitate routinization' 



Degree of 



Type of 



Core 



Minimal 



Strong support for innovation by; No 

Adminis- Practi- Active adversary 



routinization/site b innovation application competition Payoffs 0 Prior need Clients Community trators tioners innovator group 



High 

Indianapolis (10) 
Birmingham (9) 
Dallas (9) 
San Diego (9) 
Nashville (8) 
Miami (7) 
Boston (7) 
Cincinnati (7) 

Moderate 
Akron (6) 
Dallas (6) 
Rochester (6) 
Omaha (5) 
Tampa (5) 
Memphis (4) 
Portland (4) 

Marginal 
Oakland (2) 
Omaha (2) 
Rochester (2) 
Denver (0) 



PC 

MICU 

CAI 

CAI 

PC 

PC 

PC 

BT 



BT 

MICU 

J-A 

J-A 

CAI 

BT 

CCTV 



CAI 
CCTV 
CCTV 
MICU 



Source: Yin et al. , -1978. 

a The presence of these factors is indicated by a plus (+) symbol in the table. 
b The routinization score for each site is shown in parentheses. 
C|, Perceived" payoffs, not the result of any evaluation. 
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HIGHLY 
ROUTINIZED 



to 
0 



MARGINALLY 
ROUTINIZED 



/ 



/ 

Displaced 
Practice 
(if relevant) 



/ 



V 



/ 



\ 



/ \ 



V 



/ 



\ 



/ 



\ 



/ 



/ 



(3) 

Disappearance Stage 

Governance rules change 
Internal and permanent 
practitioner training 
Promotion of personnel 
Turnover in key personnel 
Attainment of widespread 
use 



1 



/ 
\ 



\ 



/ 



/ 



r 

/ 



/ 



/ 



/ 



Ex pansion Stage 

• Equipment turnover 

• Transition to local 
budgetary support 

• Appropriate organiza- 
tional status 

• Stable arrangement for 
maintenance and supplies 

i Personnel classification 
and certification 

TT 



/ 



(1) 



Improvisation 
Stage 

No passages or 
cycles achieved 

I 



Innovation is operational 
Support by innovator 
- Decision to adopt 



\ 



\ 



\ 



\ 



\ 



\ 



\ 



Top agency officials continue 
to allocate resources \ 



Coordinator works directly 
on organizational changes 

Innovation is applied to 
core practices 

Practitioners gain direct 
experience with innovation 



\ 



Public (if aware) 
identifies innovation 
as part of standard 
practice 

Other jurisdictions 
become users of 
innovation (and may 
purchase service) 

Old practices are 
discontinued 

- Practitioners derive 
clear benefits and 
fully support innovation 
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Figure 7, Routinization process (Yin et al., 1978). 



Table 9 

Relationship of Army Regulations to Model Activities 



Analysis of Research, develop, test, £ Disseminate Institutionalize 

Activities requirements evaluate solutions information the findings 



HRN Progress Reports (AR 70-1) ARDIS Utilization Report (AR 70-8) 

STOG Meetings (AR 70-8) SSTI (AR 70-45) 

Indicators DoD Program (AR 70-8) MIS (AR 70-1) 

TAS TAS (AR 70-8) 

QRI/QRRI (AR 70-35) HFRDC (AR 70-26) 





ARI 


ARI 


ARI 


Sponsor 




Contractor 


Contractor 


DCSRDA 




Agencies 


DCSRDA 


DARCOM 


DARCOM 




Involved 


DCSPER 

FORSCOM/USAEUR , 

' Sponsor 

TRAD0C 


Sponsor 


Sponsor 





Oversight/ Scientific Advisory 
Audit Boards Board & GAO 
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Table 10 



Proposed Classification Scheme to Document Activities During the Institutionalization 

of a Training Product 3 



Utilization 



Implementation 



Policy 



End of 6.3A support 
Utilization Reports 



Passages 



w 



Transition to support by local 
agency funds 

Establishing of appropriate or- 
ganizational status 

Supply and maintenance provided 
by agency 

Training product functions be- 
come part of job description 
or prerequisites 



Use of training product becomes 
part of regulation/ manual, 
etc. 

In-house training program devel- 
ops skills to use training 
product, which becomes part 
of professional standards 
and curriculum 



Cycles 



Survives equipment turnover 



Promotion of personnel ac- 
quainted with the training 
product 
Turnover in key personnel 
Attainment of widespread use 
(standard practice) 



Based on the work of Yin et al., 1978. 
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Recurrent Factors to Consider During Training Technology Transfer 



Two major classes of factors recurrent throughout the whole training 
technology transfer process are sponsorship and self-renewal capability. 

Sponsorship . The authoritative support and/or active proponent of a 
training product and necessary considerations influencing the successful 
incorporation of an innovation sponsorship may be viewed on two levels. 

1. User. Initially, highly authoritative policy makers are needed to 
promote the technology transfer process. Researcher interaction with all 
levels of users should follow, however, to insure the endurance of this 
process to completion (i.e., acceptance and effective use of the training 
product by the user). That is, successful incorporation of the training 
product will depend more on within-unit sponsorship as the technology trans- 
fer process proceeds. At the outset of a project, active proponents should 
be sought and encouraged, such as a sponsor's representative who would work 
closely with the researcher throughout all the activities of the training 
technology transfer process. This linkage agent role for the user would 
help to minimize current and subsequent resistance from the eventual "front- 
line" user (i.e., those at the implementation and receiver levels of users). 
For example, the sponsor's representative could be given a temporary-duty 
status or assigned "contact" days to work end consult "in-house" with the 
researcher. During this arrangement, the representative could deliver or 
arrange seminars designed to explain the training product to prospective 
user agencies. Pros and cons could be discussed at these meetings, details 
negotiated, and feedback provided for incorporation into the RDT&E activity. 
Thus, user resistance encountered during these sessions would be recognized, 
discussed, and perhaps resolved prior to full-scale dissemination of the 
training product. Therefore, the sponsor's representative would act as a 
coordinating influence, translating "front-line" user concerns into useful 
RDT&E considerations. 

2. Researcher. It is the researcher's responsibility to determine the 
technical feasibility of satisfying the user's requirements. Sponsorship 
within this framework focuses on the RDTSE support given to a training prod- 
uct. Collocation of researchers with uisers , such as hRl field units, is one 
way in which this support can be perceived more quickly by the user. It is 
suggested, however, that the role of field units be expanded to include re- 
sponsiveness to regional (vs. local) Army needs, with major and minor research 
efforts allocated jointly by the field unit and ARI Headquarters. Similar 
efforts have been demonstrated by the Navy's Field Engineering Offices, which 
have resulted in greater timeliness in response to user requests for innova- 
tions. It is realized that, in the extreme, collocation could result in 
parochialism and unresponsiveness to the global needs of the Army (Sands & 
Glaser, 1978). However, with these considerations in rcind, the lag in im- 
plementation of training technology R&D findings has been shortened when 
RDT&E has been collocated with the user (Alluisi, 1977). 

Self -Renewal Capability . This factor refers to those- variables which 
help the~user to incorporate and maintain the training product after researcher 
intervention. Several variables are relevant here: 
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1. Training , Introduction to and on-the-job training in the use of 
the training product for both administration and implementation levels of 
users should be initiated no later than the dissemination of information 
activity. Effective training programs should be coordinated with the re- 
searcher and user (i.e., at this stage, the training schools would be con- 
sidered a "user" also) , with the ultimate goal of within-agency (in-house) 
training programs that would prepare subsequent new personnel for using the 
training product. In~house training programs, thus, promote both initial 
competency as well as endurance of the use of the training product. 

2. Resistance . As implied earlier, resistance to a novel training 
product is a normal phenomenon. However, resistance may also be viewed as 
an indication of a lack of researcher-user cooperation to prepare "front- 
line" users for the outcomes of the training product. Resistance can be 
lessened by (a) early researcher-user involvement in the training technology 
transfer process, (b) the sponsor's representative serving in an active 
linkage agent role (i.e., marketing and persuading prospective agency users), 
and (c) suitable training prior to the incorporation of the training product 
within a unit. 

3- Money . Budgetary considerations should be planned and evaluated 
during the analysis of requirements activity. Fiscal obligation for sup- 
porting the institutionalization of the training product requires strong 
sponsorship at the policy and administration 1 evels of the user. There is 
no use in directing efforts toward user acceptance if subsequent institu- 
tionalization of the training product will not be funded.. 

4. Personnel Turnover . This concern is important in the Army because 
of frequent tour-of-duty changes (every 2 to 4 years) . As mentioned earlier, 
a good in-house training program should lessen personnel turnover problems 

in maintaining the use and incorporation' of a training product. Also sug- 
gested is an extended tour of duty for those key personnel involved in the 
initial utilization of a high-priority training product. Such an extension 
may provide the necessary foundation for the subsequent implementation and 
standardization of the training product within the unit. 

5. Reward (Incentive) System . A set of procedures (regulations) should 
be instituted that provides incentives for researcher-user efforts in train- 
ing technology transfer. For example, both implementation and receiver levels 
of the user could be given official recognition/awards, cited in their per- 
sonnel records for promoting/using the training product in an effective manner. 
Both policy and administrative levels of the user could be given more budget- 
ary and policy leeway to promote training technology transfer (i.e., such as 
being allowed to spend money saved in implementing the training product, in 
other areas given high priority by the local user agency) . Other possible 
incentives are accelerated promotions and/or seniority, choice of tour of 
duty, choice of training programs, or monetary bonuses. 

Possibilities for an Evaluation Scheme 

As presented earlier, monitoring, evaluating, and feedback occur through- 
out the entire training technology transfer process. These activities may 
be in the form of emergent requirements, changes in specifications during the 
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RDT&E effort, validation feedback reports, or research utilization predic- 
tion models. If given the funding controls, follow-up feedback evaluations 
should also be conducted by the researcher (outside-agency) on an annual 
basis, while the user (wi thin-agency) should conduct in-house evaluations 
more frequently (i.e., semiannually). The specific ways of conducting such 
an evaluation are lacking; however, the following format is suggested for 
use. 

To answer definitively the question of how many training products are 
in fact used, a systematic approach must be formulated. First, the types 
of training products to be evaluated must be defined. One could start by 
using the classification scheme reported by Drucker (1977) . Second, the 
events that accompany the acceptance and use of a training product must be 
clarified and documented for data collection purposes. The passages and 
cycles classified under the stages of the institutionalization activity 
(Yin et al., 1978) could serve initially as the critical "utilization" 
events. Both of these suggestions are presented together in matrix format 
in Table 11. 

If this scheme is used, certain passages and/or cycles may not be appropri- 
ate to certain training products, and thus, modifications will have to be 
made on an individual basis. This kind of approach would provide systemati- 
cally obtained information on whether or not a utilization problem exists, and, 
if so, the degree to which this problem extends into the institutionalization 
activity, and for what products. Additional information needed to address 
this problem is proposed in Appendix G. 

A Proposed Modification of Current RDT&E Funding Categories 

As mentioned previously, materiel system RDT&E categories are used in 
training research RDT&E. These categories essentially direct the flow of, 
and diffuse responsibility for, training technology transfer. Unfortunately, 
neither near-term nor long-term training R&D efforts can be isolated into 
discrete operational control afforded by the current RDT&E categories. Even 
well-planned training research involves (within-category) serendipitous find- 
ings which promote reflection on and modification of ongoing research efforts. 
It is suggested, therefore, that it would be desirable to modify the present . 
funding categories. The objective of this proposal is twofold: (a) to pro- 
vide some degree of fiscal and operational control by the research community 
over the institutionalization of training products, thus providing the re- 
search community with the capability of sharing the accountability for the 
acceptance and effective use of the training product, and (b) to provide 
timely and well-planned training research programs that support Army decision- 
makers for both near-term and long-term problems. 

Specifically, this proposal suggests an incorporation of 6.1-6.7 funding 
categories into two research funding activities administered by ARI. This 
suggestion is outlined in Table 12. The primary proposed modification to 
the present RDT&E funding system, as stated above, is the full-scale opera- 
tional and budgetary control inherent in the Programmed Fund. This control 
promotes acceptance and effective use of a training product. The Programmed 
Fund is designed to provide a well-planned effort to improve training tech- 
nology transfer. This effort is realized in an "Implementation Plan" which 
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Table 11 



Proposed Classification Scheme To Measure "Product Utilization" 
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Note: Training product descriptions extracted from Drucker, 1977. 

a p = passages, C = cycles? specific descriptions extracted from Yin et al., 1978. 



Table 12 



Proposed ARI Research Funding Activities 



Analytic fund Programmed fund 



(30% - 35% budget) 

1. Combines previous 6.1 and most 
6.2 monies 



2. Research directed toward provid- 
ing input for both near-term 
and long-term problems 

3. Research initiated or provided 
by: (a) spinoff ideas from 
programmed fund/ (b) contracts 
(basic research) , (c) in-hcuse 
generation (ILIR) , and (d) in- 
house problem definition re- 
search for Army training (some 
previous 6.2 work) . 



(65% - 75% budget) 

1. Combines some 6.2, all 6.3A, 
and adds (as appropriate) 
6.4-6.7 money for extending 
operational control to 
institutionalization 

2. Research directed toward solv- 
ing mainly near-term problems 
(5 years) 

3. Research initiated from the 
top down (by management) , or 
laterally (from the field) . 
Research funding requires an 
"Implementation Plan," in 
which researchers and users 
document the accountability, 
audit trails, and evaluation 
schemes for each of the follow- 
ing activities: (a) require- 
ments analysis, (b) RDT&E 
solutions, (c) dissemination of 
information, and 

(d) institutionalization . 
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constitutes much "front-end" analysis prior to RDT&E activity. If prior 
consideration is given to, and allowed for, dissemination and institution- 
alization problems (given adequate funding control) , increased likelihood 
of the acceptance and use of the training proc acts may result. Appendix H 
provides a suggested format for an implementation plan to guide RDT&E efforts 
in the Programmed Fund. Finally, the Analytic Fund would provide both the 
technical data base and problem-definition guidance for supporting research 
planning in the Programmed Fund. Whereas the Programmed Fund would see a 
training research project through the requirements analysis, RDT&E, dissemi- 
nation, and institutionalization activities, the Analytic Fund would operate 
primarily within the RDT&E domain. It is hoped that this type of funding 
framework will provide greater researcher involvement in the control over 
the acceptance and use of a training product. Given the funding authority, 
the researcher may then feel somewhat more justified in being held account- 
able for obtaining objective measures of user benefits. 



SYNOPSIS OF ISSUES IN TECHNOLOGY TRANSFER 

Definitional Issues in General 

1. Technology transfer is a process by which existing research knowledge 
is transferred operationally into useful processes, products, or programs 
that fulfill actual or potential public or private needs. The user can be 
defined in two dimensions: organizational sector (civilian and military), 
and the functional level of involvement in the Army technology transfer 
process (scientific/informative review, policy, administration, implementa- 
tion, and receiver) , These dimensions are depicted in a User Identification 
Matrix (see Appendix B) . 

Modeling and Transfer in General 

1. Development of a systems model and organization of regulations to 
reflect the flow of technology transfer within an agency. Four major activi- 
ties constitute or define human resources technology transfer in the Army: 

• Analysis of requirements (e.g., needs assessment, resulting in a 
researchable question) ; 

• Research, development, testing, and evaluation of solutions (RDT&E, 
resulting in a research product) ; 

• Dissemination of findings (can result in user acceptance) ? and 

• Institutionalization (starts with the utilization of the product by 
the user and eventually is incorporated within the user's agency as 
a policy matter) . 

2. Between- Versus Within-Technology Transfer (see Chapter 1). 



ERLC 



40 $3 



Specific Needs Assessment 



1. Lack of systematic methodology to assess the needs of the military 
user prior to the initiation of R&D . Needs assessment should consider 

(a) researcher-user interaction to arrive at a consensually defined research- 
able question, (b) program analysis techniques to define and weigh alternat- 
ing research approaches, (c) cost-benefit analysis to include subjective 
estimates of utility of a particular research proposal, and (d) individual, 
innovation, organizational, political, and economic variables which would 
influence the formulation of an R&D plan. 

2. The extent tc which dissemination and R&D implementation plans should 
be incorporated in needs assessment. Assuming that future funding for a 
particular R&D product is predictable, one could plan, prior to RDT&E, how 

to disseminate information about the R&D plan, as well as how to transfer 
responsibility from the researcher to the user in line with the shift of 
operational funding control during institutionalization of the R&D product 
(e.g., shifting from 6.3 to 6-4-6.7 funds). The analogy is planned compari- 
sons rather than post hoc analysis. 



Specific RDT&E 

How appropriate are the current funding categories/processes (e.g., 
6.1-6,7) for human resources R&D activity? It may be desirable to consider 
modification of the present funding categories. Specifically, this proposal 
suggests an incorporation of 6.1-6.7 funding categories into two research 
funding activities Osee Table 12) . 



Dissemination 

1. Procedures and factors involved in dissemination of information 
about a research product. Dissemination occurs after RDT&E and ends in user 
acceptance . 

2. Classification and documentation of the events surrounding the 
transition between researcher responsibility and user institutionalization 
of a research product. Congressional scrutinizing of military training bud- 
gets makes the research community justify their existence by demonstrating 
effective use of their training products by the target user. On a research 
level, this goal is translated into developing objective measures of user's 
benefits. On the other hand, however, the amount of fiscal control lent to 
the human resources R&D community generally stops at the 6.3 funding category 
officially, and further control of the research product is taken over or dis- 
persed by other agencies from the 6.4 to the 6.7 funding categories. This 
procedure is a carry over from weapon-systems 1 development practice in which 
software (e.g., training) development is made equivalent to the hardware 
(e.g., materiel system) development process. Therefore, if the human re- 
sources R&D community is to be held accountable for the acceptance and ef- 
fective use of their training products, a significant amount of budgetary 
control over the dissemination, utilization, and implementation of their 
training products should be allocated to the HR RDT&E community. At present 
Congress is asking the HR RDT&E community to be responsible for the operational 
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effectiveness of their training products over which the human resources R&D 
community has no formal budgetary control. 

3. Linkage agents and change agentry effects on user acceptance of 
research products. A linkage agent role of the sponsor's representative 
would help to minimize current and subsequent resistance from the eventual 
"front-line" user — those at the implementation and receiver levels of users 
(e.g k/ trainers and students). 

Institutionalization 

1. Documentation of the critical events during the institutionaliza- 
tion of a research product. 

2. Determining the "time line" of a research product's incorporation 
by a military user. 

Prediction , Monitoring, Evaluation , and Feedback 

1. Predicting the successful incorporation of a research product by 
the user. Several models have been developed to predict the outcome of a 
research product; this approach attempts to translate descriptive factors 
into numerical values that can be incorporated into a mathematical equation, 
whose result would provide an index representative of the predicted success/ 
failure of the acceptance/use of a research product by the user. The Linker 
Model's main contribution is the recognition of both procedural (formal) and 
behavioral (informal) factors influencing successful dissemination of infor- 
mation about a research product. 

2. Developing a prediction methodology in-house. To answer definitively 
the question of how many research products are actually used, a systematic 
approach must be formulated (see Appendix G) . 

3. Interorganizational/agency involvement. The information depicting 
those DOD agencies that spend RDT&E funds, and the types of RDT&E funds used 
is a matrix consisting of 6.1 through 6,7 funds listed along one dimension, 
and the names of the agencies along the other dimension. The question of 
interest is f rom-whom-to-whom are funds handed off during the transition of 
RDT&E funding control of the research product. This information would indi- 
cate the extent of desirable interaction with specific arencies involved in 
the handoff of budgetary and operational control of the research product. 
(For example, one could see how many agencies spend/use funds during the 
6.3-6.4 interval.) This matrix would seirve as a "funding flow" chart con- 
sisting of clusters of agencies which may be correlated with organizational 
events critical to tiie outcome of the research product. 

4. Determining baseline success rates. Success rates of research prod- 
uct utilization would help to guide subsequent funding and research directions. 
Determining the baseline rate has not been addressed empirically. The defi- 
nition issue could be resolved by using both objective and subjective sources 
of information, as mentioned previously. Then the scope, to be addressed, 
could be limited up to 6.3 (user acceptance) , or could include the transition 
period (6.3-6.4 interval) of budgetary and operational control. 
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5. Monitoring, evaluation, and feedback. This activity goes on during 
all the main activities of the model. Once a research product is defined as 
standard practice (policy or routinization) , monitoring and evaluation of 
this practice can result in a new emergent requirement, which in turn initi- 
ates the training technology transfer process. 

Recurrent Issues 

Recurrent throughout the training technology transfer process are spon- 
sorship and self-renewal capability; incorporation of an innovation sponsor- 
ship may be viewed on two levels: user and researcher. 
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APPENDIX A 



DOCUMENTATION OF INFORMAL COMMENTS AND ANECDOTAL EVIDENCE 



D ocument 1 (Shields , 1976) 

1. In one unit, in the FORCES COMMAND (FORSCOM) SCOPES (an .engagement 
simulation system for the infantry) was introduced, briefly used, 
and for the past year has simply sat idle. 

2. The Training Aids and Audiovisual Support Services Organization 
(TASO) at another FORSCOM installation indicated that a game de- 
signed to assist tankers in learning combat tactics had not been 
checked out since its arrival. 

3. A tank-driver trainer-simulator was found sitting in a warehouse — 
simply occupying a great deal of space — not training. 

Document 2 (Sands & Glaser, 1978) 



1. Army Case Study— The Value of Stereoscopic 
Viewing in Image Interpretation 

The research objective was to assess the usefulness of stereoscopic view- 
ing within serial surveillance systems in terms of quality of information ob- 
tained and the rate at which it is extracted. Both tactical and strategic 
types of interpretations were used in setting up performance measures which 
were administered to two matched groups of image interpreters. For each 
measure , stereo pairs were provided to one of the two groups and no.istereo 
photographs to the other. Data were analyzed by comparing mean scores through 
t-tests and analysis of variance. 

Stereo viewing and nonstereo viewing of the tactical and strategic photo- 
graphs were found to be equivalent in terms of the quality of information pro- 
vided and confidence expressed by interpreters in the information they ex- 
tracted. Accuracy and number of targets identified under the two methods of 
viewing were similar; that is, no statistically significant differences were 
found . 

This research was suggested by Army researchers after they had been 
called in by the Air Force to consult on a similar problem. Findings in the 
Army research then suggested strongly that the value of stereo viewing should 
not be taken for granted, and, in fact, led Army researchers to suggest that 
the need for the stereo capability should be clearly demonstrated before new 
display equipment with stereo capability is developed for use of interpreters 
in detecting and identifying militarily significant objects. Several other 
research efforts by other organizations corroborated these findings. 
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but Army researchers found a strong existing conviction in favor of 
stereo and a system already highly geared up for obtaining and interpreting 
stereo imagery. The research had little impact on decisions to build more 
stereo capability, despite consistent replication of the above and the con- 
duct of several projects on overlapping imagery (60% is needed for stereo) 
that indicated conditions of nonoveriap (except the small percent needed to 
insure 100% coverage) to be superior in time to interpret and no different 
in terms of accuracy or completeness of interpretation. 

The practical aspects of stereo coverage argue strongly against it. In 
addition to the ext T a 30% to 40% in time required for stereo viewing by the 
interpreter mention d above, there are several severe systems costs involved. 
It takes twice as many photographs (and processing) to cover a given area 
using stereo. Dollar cost as a function of the number of systems using 
stereo is readily derived. It also means that twice as many reconnaissance 
missions need to be flown (again one can compute dollar costs) and twice as 
many planes and lives risked. 

And still systems users have persisted in using stereo. 

2. Army Case Study — REALTRAIN 

REALTRAIN is an improved, low-cost training and evaluation technique 
for use in Army tactical training exercises for combat units. Realistic, 
two-sided free-play tactical training employing recognized principles of 
learning is achieved through simulated combat engagements. 

Originally this research sought to develop a method for evaluating 
individual tactical performance under simulated battlefield conditions. It 
was felt necessary first to construct job situations that would demand that 
a man act as he would be expected to act on a battlefield. It was, in retro- 
spect, not surprising that the environment developed for testing became a 
powerful vehicle for training. The rationale which provided the basis for 
the initial development of methods for simulating the combat environment 
with a high degree of psychological fidelity led directly to the REALTRAIN 
method for tactical training. 

Before a research organization recommends implementation of a new train- 
ing method it is accepted practice to determine empirically whether the new 
method works and how well it works (the degree to which training objectives 
are achieved, the nature of the skill acquisition curve) and to compare the 
new method with the method it was designed to replace. This was not done 
with REALTRAIN . The decision was made by the Training and Doctrine Command 
(TRADOC) to implement the method before the standard validation procedures 
had been conducted. 

The reasons for x'RADOC's decision were (a) the heavy cost of conducting 
an evaluation of a new unit training technique in the field, (b) the rapid 
and enthusiastic acceptance of the method by troops and commanders, (c) the 
overwhelming face validity, and (d) the fact that no technique for realistic 
tactical training had previously existed. 
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The REALTRAIN training method was implemented by a TRADOC Mobile Train 
ing Team (MTT) during the period 3 November 1975 to 5 March J.976 at four 
divisional trai-iing sites throughout the U.S. Army Europe (USAREUR) . The 
.implementation in USAREUR afforded an opportunity to conduct research which 
could be used to improve bacbical training and evaluation techniques furthe 
in an engagement simulation context, specifically providing a valuable em- 
pirical ba^e and data source for the analysis of tactical performance by 
participants in the exercise, participant and controller reactions to this 
new method, and the cost of conducting such exercises. 

That implementation by TRADOC involved a cor-t of several million dol- 
lars. REALTRAIN is still not bei^g effectively utiJ.ized today for a number 
of reasons: 

1. REALTRAIN has evitailed very significant — even revolutionary — 
changes from the ways of past training and it was almost too big 
a challenge to do it well. 

2. One specific resource requirement that has given heartburn has 
been the requirement for controllers who are required for conduct 
of an exercise, though controllers learn as much as the trainees. 

3. Young, inexperienced officers don't like to conduct exercises and 
lose badly, as many of them do, even though the learning experi- 
ences are invaluable. By the same token, bo be a participant 
casualty through inappropriate performance in the exercise is 
also stigmatic. To help solve th'i young officer problem, a leader 
board game has been devised to get the officers better prepared 
for the exercises. 

4. Logistical requirements are overwhelming — training aiumo is expen- 
sive, tactical radios to support control of the exercise are hard 
to get. 

TRADOC officials have been much aware that initial successful utiliza- 
tion of REALTRAIN may rest critically upon keeping researchers involved in 
the handing-off process to help solve some of -,;he problems of utilization. 
The purpose of a new program is to do just that— have researchers assist in 
preparing a method for implementation and observe problems of utilization 
and help make refinements, periodically return to reassess utilization 
procedures . 

« 

Comment: Normally the early grabbing of a research product for imple- 
mentation before the completion of research is considered in the nature of 
eating the bean sprouts, intended for planting to ease famine. In this 
instance, however, researchers did not consider that hann had been done to 
the ultimate utilization of this product, especially in view of the fact 
chat implementation aided research. 

3 . Army Case Study — Armor School 

The Armor School, in conjunction with Naval Training Equipment Center 
(NTEC) , contracted with General Dynamics to develop a Miniature /Armor 



Battlefield (MAB). The MAB had radio-controlled tanks on i miniature (6 f ) 

field, with TV sensors in the tanks and hit/kill sensors; it was intended 

for the simulation of platoon versus platoon engagements. Development cost 
was approximately $1 million. 

The Armor School asked HumRRO to evaluate the system and to wcrk out 
training procedures. The research group had been working on a simplified 
version of the system which eliminated some problems encountered with the 
TV sensors on the General Dynamics device. The evaluation was performed 
with a 24-hour field exercise as the performance criterion? the system was 
shown to be effective for training tank crews. A follow-up in Europe using 
commanders' ratings as a criterion resulted in higher ratings for the sys- 
tem than for conventional training techniques. 

The Armor School recommended that the MAB and another research product, 
the Armor Combat Decisions Game (CDG) , be developed by TRADOC. Regulations 
were published governing the use of these training devices. NTEC was to 
have action on obtaining the devices, in simplified form, per recommendations 
of HumRRO. NTEC tried, however,- to improve the tank model further but en- 
countered problems with miniaturization of alecfronic components. A con- 
tractor could not be found to build the devices to specifications; a later 
attempt by the Training and Doctrine Command, Deputy Chief of Staff for 
Training to get the devices built through Naval Training Device Center 
(NTDC) failed as well. 

At present the CDG is in use by the? Canadian Army in theii own version, 
successfully. The MAB has not yet been produced as a training device. An- 
other product/ a map board which was part of project RECON for Armor train- 
ing, was given to NTOC, formerly NTEC , for development. It i^ currently 
confined in use to the Ohio National Guard. 



In February 1966, the Assistant Secretary of Defense for Manpower and 
Reserve Affairs requested research on a common aptitude batcery that could 
be used by all the services in the high school testing program. The Army 
was designated lead service to determine to what extent the aptitude tests 
of the several services were interchangeable and to develop an appropriate 
test battery. The Armed Services Vocational Aptitude Battery (ASVAB) , con- 
sisting of a common core of abbreviated forms of tests found to be inter- 
changeable, was a first product of this endeavor and was put into use to 
test potential recruits in the last year of high school. 

As the original research called only for the development *of tests for 
the high school program, only tho^e t-^ts common across all services were 
considered. Thus, the sen vice wit.h the smallest set drove the system. 
1'i.om School Year 1968/1969 ASVAB Form 1 was used in high school testing; 
in 1972 Forms 2 and 3 wero developed for high school testing, and they also 
became the operational batteries of the Air Force a>:0 Marine Corps. Army 
bowed out as Executive Agent for ASVAB research and Air Force took over. 

• In the iuidale of 1974, the ASD (M&RA) decided that as of 1 January 1976 
there would be a single classification battery, ASVAB, to serve the primary 
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Army Case Study — ASVAB 
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selection and classification purposes of all the services, as well as for 
high school testing. The new requirement represented an important change 
of concept. To produce one selection and classification battery to serve 
needs oi all services, t'la service with the largest set of requirements 
drove the system, and L'jus, a 13-test battery was necessary. 

The battery was fielded 1 January 1976, but with the short lead time 
available, it was done with no validation, a fact which some Army observers 
believe contributed heavily to the large attrition rates in TRADOC schools. 
In addition, norms have had to be adjusted and are still being questioned 
in the Army, Navy, and Marine Corps. 

This case study depicts the development of a product for which sponsor 
interest, enthusiasm, andlmpatience are factors that have to be dealt with 
to delay (rather than hasten) utilization, so that the product will have a 
reasonable chance to be effective in operation. 
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APPENDIX B 



DEFINITIONS OF USER, RESEARCHER, AND TRAINING PRODUCT 



Us er Definition 

The user is defined in two dimensions: organizational sector (civilian 
and military) , and the functional leval of involvement in the Army training 
technology transfer process (scientific/informative review, policy, adminis- 
tration, implementation, and receiver). These dimensions are depicted in 
the User Identification Matrix, which follows. Note that the matrix is not 
all inclusive. It serves as a starting point to help determine what strategy 
is needed to facilitate training technology transfer based on the user's 
level of involvement and sector in the process. For example, attitudes and 
factors related to the training product , special group px. < :::sses , and the 
organizational structure are, important variables influencing the acceptance 
of an innovation (e.g., Havelock, 1976, HIRI, 1976). Knowing where a par- 
ticular user is located in terms of sector and level of involvement will 
provide information on the user's perspective on attitudes and factors re- 
lated to the aforementioned variables. If one were interested in congres- 
sional subcommittee support, then the user's perspective may be couched in 
terms of cost-effectiveness of the training product, political liaison, and 
organizational contact via a congressional staffer. Or a sponsor's repre- 
sentative may be interested in the ability of Mia training product to 
satisfy the sponsor's specific requirements, would be in contact with the 
researcher on an information and interactive basis with the researcher, and 
would function as an organizational link to promote change agentry with the 
organization to which the training product is directed. It is realized 
that there may be some, degree of overlap when categorizing users into levels 
and sectors, thus the user rate within this matrix represents an emphasis 
other than an inclusive/exclusive class: Irication. 

« Key Words 
Sources — ^ 



Drue* *£, 1977 Adoption agency 

Havelock, 1976 Chan 9 e a ^ ent * 

Client 

Customer 

Linkage a«ent* 

Receiver 

Sponsor 



*I have narrowed the definition of these terms to the user only (see Have- 
lock for expanded definition) . 
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User Identification Matrix 



Researcher Definition 



Sector' 



Military 



Levels 



Scientific/ 
informative 
review 3 



Policy 



Administration 



Implementation^ Receiver 6 



Members of 


Congressional 


scientific 


subcoEiittees 


i . i community 
Civilian 


ft 


Project 


SE0DEF 


manager 


SECArmy 



ARI DCSRDA 
Other DOD DCSPER 
scientists TfflDOC 



Sponsor's representative Instructor Student 
Training manager T;ainer Trainee 

Commandant Operations NCO Soldier 



The researcher is defined in two dimensions: organizational sector 
(civilian and military), and the functional level of involvement in the 
RDTSE of the training product (application and characteristics). These 
dimensions are depicted in a Researcher Identification Matrix, which fol- 
lows. This matrix represents the primary functions of the researcher (s) 
in response to user requirements. An application function would involve 
translation of user mode into a researchable question and the monitoring 
of the RDT&E activity to insure that a solution is found. Characteristics 
functions underlie the actual RDT&E effort to produce the training product. 



Sources 

Drucker, 1977 
Havelock, 1976 



Key Words 

Contract/grant monitor 
Contractor/grantee 
Developing agency 
In-house research director 
Program/project director 
Researcher/bench scientist 
R&D manager 



Researcher Identification Matrix 



Dses information for professional review. 



b 

Creates doctrine and budget levels. 
Executes doctrine and uses the budget, 
^fenages, operates, and/or maintains the research product, 

g 

Uses the product as a beneficiary. 



Levels 



Sector 



Application 



Civilian 



Characteristics 



Contractor 



Military 



Developing agency 
ARI 



ARI 



Monitoring and evaluation of RDTSE activity to insure fulfillment of user 
requiiements. 

Conduct "hands-on" research :tivity to find solution to researchable 
question. 
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I/raining Product Definition 



There is a strong agreement (£ = .82) between researchers and users in 
what they consider to be a research product, although a greater percentage 
of researchers (72%) check more items qualifying as research products than 
uj users (54%) (Drucker f 1977). Of those items considered, an average of 
77% of both researchers and users classify training products as research 
products. While a large percentage of both researchers (71%) and users 
(74%) consider training programs and devices as examples of training prod- 
ucts, there is disagreement with respect to classifying training literature 
(e.g., manuals, circulars, instructions). That is, 85% of the researchers, 
and only 53% of the users, consider training literature as a research prod- 
uct (Drucker, 1977). Based on these data, Druckcr (1977) states that, 

" the user does not give enough credit to research in helping him with 

his implementation of policy changes or may not accept that the researcher 
makes a contribution with such products as handbooks and manuals." It is 
suggested that in order for a training product to be used effectively, the 
user must be shown (convinced) how the product can be integrated, used, and/ 
or maintained within one's operational setting. Training programs and de- 
vices possess considerable "face-value" structure which users can appreciate 
initially in fulfilling their training needs. However, training literature, 
as a "stand-alone" product, lacks sufficient integration to solve training 
needs from the users 1 point of view. Optimally, a training product is de- 
veloped to effect a change. Differences in viewpoints as to what specific 
products accomplish this objective can be lessened if the researcher and 
user form a continuous, critical dialog during the training technology 
transfer process. 
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APPENDIX D 
ANALYSIS OF REQUIREMENTS ACTIVITY 



r. . . •: (all references to AR 70-8 
sta 9 e Technique aiiless otherwise noted) 



1, Decide to plan 

e Identify users and their functions 
o For proposed training products, 
consider following questions: 

a. Will proposed product or 
change in system reduce 
costs? 

b. Will product result in bet- 
ter results? 

si 

u Cp Is the problem urgent? 

d. Will "management 11 be recep- 
tive to proposed 
undertaking? 

e. How large is the project and 
how long will it take to im- 
plement? Will the results 
be lasting? 

f . Can product effort and con- 
version costs be justified? 

g. Is the current staff capable 
of executing this project? 
Will additional personnel 

be required? 

h. Is product feasible with 
existing hardware/software? 



a. Group decision of users 

b. Group presenration/conse:.' us 
d. Representative panels ba:: i 

on users inv:lvement/aut;:,.:ity 
d. Interviews 



"■.1: DCSPEE reviews/approves annu- 
i, 5-2 portions of the Science 
7 program submitted by ARI; 
:, is , AR 70-1, 6.1 known as Single 
x^r . . :ding (SPF); work units known 
: 5::::;r.:ic areas; 6.2 known as Single 
■ '.-r.»r Lament Funding (SPEF); work 
:i::t. as Technical Areas) 

:T"I-E: iordina:es troop support re- 
ii: ,i xnt£ between appropriate major com- 
distributes ARI 5-year plan annu- 
• 7 "o staff agencies and major 

: " jM/L" "AERU : Sponsor RDT&E projects 
ince with their responsibility 
c\t readiness. 
V 1 /'/ ■ Defines immediate *nd long- 
■ training objectives, combat devel- 
:s„ and strategies which require 
development. 



1. ^ 
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.mae 



2. Identify prr sponsor i::.- ^acy 



9 Access thf :ilitv of t:. :^ni:e- 
mf ;. 3 an. .is 

a. Obta-j; .^tnorizatior; . ;. : - 
; ;ic:.'^at£ in. requirent-i.is 

b. organization*. ;n? 
■• iucnaine function 

. cf users 
c ^ :,ucz interviews/art^ ■ 
- ings to obtain ne:e^ - 
information 
::: ina interviews ad i^et- 
i. middle and high:.:: 
in: anent 
■: current systems .: .-- 

c-: :ion to the exten: 

r.i: osry to understand 
opt ..iuions of the ciirr 
sys 

Correlate information z 
relevant users 
. Cedie to continue or ■ 
investigation 
o antify domain/limitatior :or 
.arming, e.g., trainee, : a, 
: .eld-use, etc. 
a „ Identify and select techniques 
for requirements analysis 



03 
0 



n 



Technique 



Army anal:- (all references to AR 70-8 
ur.^ss otherwise noted) 



a. K: lew reports of incident: . 

occurred recently in m . 
ii context 

b. ^ iew in-house statistics : 

ciu-ncy 

c. Ee::iew current documentstii.: 
ari operating forms 

d. Record personal observation 

e. Interview relevant levels :: 
users 

f. Meet and discuss issues with 
users and higher management 



■.-it 



a. Group participation of all 
levels of users as possible/ 
feasible 

Guidelines: 

I. Outcomes of requirements analy- 
sis are stated in measurable per- 
formance terms; including all the 
criteria used in any measurable 
objective: 



L-CJSPEH: Iv, lews/approves annually the 
ARI G.3A p don of thf; program following 
a presents on of the proposed program 
10 sponsc: md other interested agencies 
zz which , • sponsors review the program 
snd reconirs needs and priorities as 



required. 

6.3A expend./! in i^ponse to a user re- 
quirements stated in: 

a. Sconce & Technical Objectives 
Gi^de (ST. VI) 

b. Hisan Research Need (HRN) ad- 
visory statement 

c. Jointly approved DOD program 
ARI: In r sparing a plan for the ad- 
vanced dev .opment portion of the pro- 
gram, ARI -ill identify the work with 
a DA- and/ jr DOD-approved project. 
HRNs which support a procurement action 
will be ir. eluded as supporting back- 
ground documentation with the total 
package w on processing Determinations 
and Findius (DSF). 
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Technique 



Army analog (all references to AR 70-8 
unless otherwise noted) 



o Criteria usually associated 
with training objectives 
are useful here 
• Assessment objectives should 
include outcomes for at 
least the receiver, imple- 
mented administrator, and 
policymaker, and should list 
objectives within each of 
these levels. 
II, Insure all "partners" in train- 
ing process are involved in selec- 
tion and decision. 

co III, Do not select techniques that 

affix blame or could be used to do 
so. 



3-, Diagnose ths problem 



o Determine the existing condition 
for all partners ("what is") 

a. User levels: 
Receiver 
Implementer 
Administrator 
Policymaker 

b. Context: 
Organization 
Social-interactive 
Resources (budget, person- 
nel/ etc.) 

c. Insure existing conditions 
stated in measurable terms 
of performance 



a. Re values of users: 

o Rucker's (1969) value 
analysis 

b. Re starting concerns/parameters 
« Sweigert's (1969) concerns 

analysis 

• Stufflebeam's (1968) CIPP 
analysis ' 

t Flanagan's (1954) critical 
incidents 

• Kaufman et ah (1954) utility 
criteria 

• Key Informant Approach (Hage- 
dorn et al., 1976) 

• Group forum 

t Structured workshops 



ARI: Will arrange for meetings of the 
principal investigator and sponsor's 
representative initially to coordinate 
RDTSE objectives and later to facilitate 
execution of the RDT&E effort. 

Technical Advisory Service (TAS) : Formu- 
late limited ad hoc consultative advice 
provided by Army scientists or contract 
consultants of the developing agencies to 
assist users/sponsors in formulating 
operational requirements in terms amen- 
able to RDTSE treatment. 
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Stage 



Technique 



Army analog (all references to AR 70-8 
unless otherwise noted) 



9 Determine the required condition 



a. Obtain consensus information 
from users 



a. All teuiniques under b. in 
stage 3 above 



b. Delphi method (Dalkey, 1970) 



b. Goals/objectives in "indi- 
cator" format/Milestones 
Chart 



d. Researcher should be aware 
of state-of-art assessment 
in training to insure required 
conditions are realistic 
within given time-frame 



c. See Table D-l 



c. Bracketing method (similar to 
Delphi, but without a category 
defining requirements provided 
initially] [see La Bay & 
Peckenpaugh, 1974]. 
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w Comments on Stage 3: 

These steps essentially involve translating needs into researchaole questions. Zaltman's (1978) Victory Model delineates 
the following factors which the researcher should be aware of when involved in Stage 3: 

» Ability to communicate among the users 
t Reward structure of the organization 
i Availability of resources for solutions 
ft Time-lag for formulating solutions 

9 Different user levels may result in different user perceptions of problems 
a Types of resistance: 

Semantic: difficult in communication 

Operational: imprecision in identification of problems 

Attitudinal: organization-specific problems in information flow among users and researchers 
(see Zimbardo 5 Ebbeson (19:0) for information on how to change attitudes] 



o Reconcile discrepancies among 
users 1 viewpoints 



Select techniques from steps in 
Stage 3 





Stage 



Technique 



Army analog (all references to AR 70-8 
unless otherwise noted) 



CO 

w 



4, Evaluate alternatives a. 

• Place priorities among the dis- 
crepancies and select b. 

a. Consider two questions when c. 
making priorities: 

What does it cost to meet the 
need? 

What does it cost to ignore 
the need? 

b. Objectives 

c. Review milestone plan 

d* Consideration of project 
magnitude 

• Estimating totality of 
meeting requirements 

• Budgeting 
e Personnel 

• Support manpower 
« Interaction with other 

agencies 
« Innovation characteristics 
o Sociopolitical trends 
I Insure requirements analysis is a a, 
continuing process 

« Types of changes to consider b. 
Requirements 
Design 

Technological 
Social/political 
Personnel/manpower 
Corrections 



Representative panel 

Delphi 

Bracketing 

Large group ballot 

Face-face consensus building/ 

ranking 



For each approved project, the developer 
will provide milestones , man-year esti- 
mates for accomplishing the effort, and 
resource requirements needed for the 
conduct of RDT&E, 



Planning/evaluation set up 
within organization 
Stated in policy that require- 
ments analysis will be updated 
and corrected periodically 



Annually, ODCSRDA updates STOG; ARI 
updates its 5-year 6.3A program* 
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FRir 



Army analog (all references to AR 
stage Technique unless otherwise noted) 



One can attempt to control change, not 
eliminate it. Estimate importance of 
change on project costs and delivery 
dates, If user (s) still want change 
incorporated, negotiate contract modi- 
fication, issue formal change notice, 
and proceed with R&D. 



oo 



Table D-l 

Requirements Analysis Summary Table 









Discrepancy 




User 




Current Required 


Level 


sector 


Identification 


condition condition 






Congress 




Policy 


Civilian 


DOD Secretary 




Military 


TRADOC 




Admi n i st rat ion 


Civilian 


DOD Army 


: 


Military 


FORSCOM/Staff 




Implementation 


Civilian 


X X X X X 




Military 


Trainer 




Receiver 


Civilian 


X X X X X 




Military 


Trainee 





This table provides a guide for documenting what is needed if the user at 
each level of involvement in the training technology transfer process (sea 
Kaufman, 1972). The table, when filled in, displays in graphic format the 
differences needed to be overcome for successful RDT&E of the training prod 
uct. Strategies could then be developed to lessen these differences by 
focusing on specific levels of users. 
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APPENDIX E 

TAEG REPORT NO. 40 (DUFF\ . MILLER, & STALEY , 1970(b)) 



Section I 
Introduction 



Background 

The benefits, costs, and risks associated with introducing technical in- 
novations in education and training can invoke large commitments of resources. 
The rational assessment of payoffs and penalties for investing in changes 
therefore deserves technical attention with the objective of arriving at sound 
decisions to accept or reject. In essence, this means making an analysis of 
the full potential range of applicability of the proposed innovation in terms 
of benefits, liabilities, and risks, including financial analysis of costs, 
and synthesizing the mass of resulting data into a decision-making presenta- 
tion. Key judgmental operations should, of course, be retained by humane. 
Explicitness in procedure and in the expression of human judgment is a key 
factor in rationality. The Educational Technology Assessment Model (ETAM) zls 
a set of comprehensive procedures and variables for this analysis, synthesis, 
and decisionmaking. Although the content of this structure, or "model," is 
directed specifically toward education and training, the structure itself is 
applicable to any rational, decision -making context. 

An innovation is broadly defined. In ETAM it is "a relatively constant 
or enduring change in the procedures, objects or functions used in any aspect 
of the instructional process which may be viewed as a benefit (or a liability) 
and has associated costs." Thus, an innovation may be a technical invention 
or it may be a structural change in the setting of instruction, such as from 
shore-based to ship-based training. In summary, innovations range across 
content of instruction, instructional procedure, student selection, and gen- 
eration and implementation of training requirements.* 

The initial ETAM study developed a complete set of manual procedures, 
parameters, and formats for all analysis, synthesis, and decision. A com- 
prehensive, descriptive taxonomy of educational technology was generated. 
Its purpose was to enable any proposed innovation to be described in a 
standardized terminology for determining the full range of potential effect 
in the Navy: students, courses, jobs, instructional devices, instructional 
development, as areas of relevance. An equivalent effort was sp- t in de- 
veloping or adapting analytic cost models applicable to the Navy's training 
environments. 



*For additional reference to educational innovations see: Miller, Robert 5 . , 
and Duffy, Larry R. , 1975. Design of Training Systems Phase II-A Final Re- 
port. TAEG Report No. 12-3, Training Analysis and Evaluation Group, Orlando, 
Fla. Chapter III. 
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T; j pioceduraJ. model also included return on investment analysis, sensi- 
tivit analysis, and scaling procedures for translating subjective evaluations 
of r<j .ive worth into utility values. These utility express! ^s became in- 
puts :: formal decision tree models to be presented to the o: decision- 
maker. A logical flow of information is maintained from one % step 
to the next, so that the assessor can readily review the sou: _ne£its 
data, cost data, judgments, and assumptions leading up to th.. presenta- 
tion of evaluation decision alternatives. The decisionmaker mq the 
input values to the sensitive parameters in the model and det = the ef- 
fect on the decision alternatives. The decisionmaker is not cted to 
working only from the formal outcomes of staff evaluations. 



Assumptions About Using the Model 

Some assumptions that were stated in the initial ETAM study should be 
repeated here. 

1. The primary user of ETAM will be the assessor of the innovation. 
He (or the assessing team) have expertise in the subject matter 
of the particular innovation to be evaluated. He has mastered 
the ETAM classification structure, at least to the level of be- 
ing able to reference its content. Furthermore, he will have had 
a~ least several dozen hours of preliminary practice in applying 
^he procedures* manually (except for calculationaJ. problems) and 
can "walk his way through" the major ETAM tasks. He will also 
have background in the operational aspects of Navy training courses, 
instructional devices and media, Navy jobs, and/or the developmental 
stages of training; at least he will be familiar with those aspects 
of these operations relevant to the innovation. This assumption 
recognizes that humans will provide the information inputs and 
judgments, whereas the model merely structures, guides, and within 
defined limits processes them. 

2. The secondary user of ETAM will be the executive decisionmaker. 

He makes the decision to commit, deny, or commute the resource for 
implementing the decision. He may question the constituent or 
summary judgments, evaluations, data sources, and predictions of 
the assessor embodied in the final evaluative recommendation. He 
can "peel back" the various layers of data and judgments entering 
into the final calculations. ETAM documentation should facilitate 
•this normal relationship between the executive and his advisory 
staff work. The service of the computer should aid rather than 
hinder this inspection. 

A key factor is the ability to identify the factors most sensitive 
to the decision outcome, and test the range jf this sensitivity 
across the limit where a recommended decision choice A changes to 
a decision choice B. 

3. The assessor will make the final judgments as to whether the inno- 
vation is or is not applicable to an entity such as a given training 
course, a given instruction vehicle, a given job or job- task. The 
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range- :>f -effect search operations, including the indexing of the in- 
novation and indexing the data base content, and automatic searches 
on data base', will facilitate those judgments, ! ut not replace them 

4. The user is not compelled to apply the procedures beyond any stage 
where common sense shows that the outcome will be hopelessly nega- 
tive. The model is segmented into stages so as to help this kind 
of efficient termination come about. It is easy to become disen- 
gaged from the model with a sensibly completed piece of work. The 
assessor is also encouraged to examine the outcome of each stage of 
work for being within the bounds of reasonableness. 

Objectives for ETAM 

The design phase of ETAM led to the specification of a set of manual pro 
cedures for assessing a proposed innovation or. change in Navy training. A 
proper follow-on objective was adapting computerized aids for reducing the 
large burdens of manual activities and supplementing rather than replacing or 
interfering with human judgmental processes. The scaling operations leading 
to expressions of utility in decision models were so important that they de- 
served, intensive study of theory and practice in behavioral utility models 
before adopting and standardizing on any given procedure. Another key issue 
was the practicability of the ETAM classification structure tc the indexing 
and searching of the content in Navy data bases dealing with training courses 
jobs, and job-tasks, instructional vehicles 1 , and media. These issues were 
the basis for the next phase of ETAM. 

The following tasks summarize the initial ETAM objectives: 

1. A study to determine appropriate scaling techn .ques which would in- 
crease the expected reliability and validity of subjective estimates 
required within the ETAM procedures.* 

2. A study to define indexing methods to provide equivalence between 
the ETAM r^nge-of -effect taxonomic elements and data base descrip- 
tors for the purpose of achieving effective data search and re- 
trieval operations.** 

3. A major report presenting a comprehensive overview of the innova- 



tions in concepts, methods, and practices that have shaped and are 
currently influencing modern instructional technology. The purpose 
was to give the intelligent layman an overview of the effects of 
innovation upon training. Such a document would aid high-ranking 
military officers and business executives in making decisions on 
applying proposed innovative techniques and/or technologies to 



*Miller, Robert B. , and Duffy, Larry R. , 1975. Design of Training Systems , 
The Development of Scaling Techniques . TAEG Report No. 32, Training Analysis 
and Evaluation Group, Orlando, Fla. 

**Refer to Appendix A of this report, TAEG Report No. 40. 
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their training system. There are parallels, but few direct equiva- 
lents, of the content of the initial ETAM report (TAEG Report No. 
12-."") . It would be an appropriate orienting background to the ETAM 
report for those using its content . 

4. Computerization of the ETAM logic so that an assessor can interac- 
tively arrive zic an accept/reject conclusion using various utility, 
probability, cost, and benefit data as model inputs. Standard 
program documentation and a user's guide will be produced as an 
output of this task. 



This section of the report has two components. The first deals with the 
procedures arising from the steps in the operational sequence of ETAM. Each 
of these steps has a set of defined information inputs, processing activities 
and defined information outputs. This procedural definition was presented in 
the initial ETAM report, but subject to modification by the results of the 
studies on scaling and on indexing procedures. The second component of the 
functional requirement is the design requirement for computer support to the 
procedures . 

Procedural Requirements 

The ETAM procedures consist of eight major tasks. The following descrip 
tions will focus on the data management components of these tasks rather than 
on how they are performed since these aspects are most relevant to the func- 
tions of computerized data storage, processing, and retrieval. 

Figure II-l is a schematic of these steps. 

Task 1 — Formalize the Description of the Innovation . A project file is 
initiated to serve as a data base for Innovation X. The innovator identifies 
the objectives of the innovation as he conceived them, target applications, 
and the results of empirical studies, if any, made from the innovation, or 
cited as relevant to the innovation. With the assistance of a staff "assess- 
or" with ETAM background, the prose description of the innovation is indexed 
according to the taxonomic classification descriptors in ETAM. The indexed 
innovation as a set of descriptors will be used, as a search specification 
against data bases in Task 5 — determination of range-of-ef feet . The infor- 
mation in prose form will be retained in a Task 1 file. 

Task 2 — Develop/Examine Alternatives to the Inn ovation. In this task 
the innovator (or other expert) is requested to consider possible alterna- 
tives to the proposed innovation which may require a lesser level of invest- 
ment funding, and possibly be more cost effective. Any outcome of Task 2 
will be treated procedurally like Task i and the outcomes of Task 1. The 
ultimate result will be to create a decision that compares Innovation X with 
the alternative Innovation XX. The profile of descriptors that indexes 



Section II 



ETAM Functional Requirements 
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Innovation XX may, or may not , be identical to the index of Innovation A. 
Innovation XX will generate its own file with a direct associative link 
to Innovation X. 

Task 3 — Make Preliminary Feasibility Profile . The formal assessment 
procedure begins at this point with a questionnaire about potential risks 
in the implementation, acceptance, and application of the innovation. The 
issues deal with organizational incompatibilities, goal/policies incompati- 
bilities, technical support requirements, funding constraints, and problems 
in attitudinal acceptance by users. A format is availahxe for entering 
risk estimates and comments on each of the key risk variables. 

If the risks are high, risk reduction projects are formulated. Their 
costs are roughly estimated and the consequent risk reduction is also esti- 
mated. If the overall risks still seem excessively high, the decision to 
reject the innovation from further consideration may be made at this point. 

A format for collecting these data is contained in Appendix B of this 
report (Figure B-l) . The content becomes part of the Project File. It will 
be used in Task 4 and appear as essential data in Task 8. 

Task 4 — Perform Analytic Feasibility Assessme nt. The results of Task 3 
are analyzed in greater depth and risk reduction studies and projects are 
further defined and costed. A preliminary decision tree is structured for 
providing initial guidance as to whether the innovation should be accepted 
outright, accepted with the additional expenditures for the risk reduction 
projects, or rejected. Presumably, a definitive range-of-ef f ect study of 
po'tential benefits has not yet been justified, or a sample of already known 
target applications for the innovation is a sufficient working basis for 
this stage of assessment. 

Project descriptions are prepared for each R&D effort with supporting 
data about resource requirements, cost analysis, and time schedules. These 
projects are grouped into packages, each of which is intended to reduce the 
overall risk tc a reasonable level. A format for collecting cost/saving 
data is shown in Appendix B (Figure B-2) . 

Decision trees are developed from estimated benefits data, cost data as- 
sociated with various supplemental projects , and risk estimations. Refer to 
Appendix B (Figures B-3 and B-4) . 

Note that at this stage, the range-of-ef feet and cost analysis has been 
only grossly estimated rather than derived from a full scale analysis. But 
even on these bases, the differences among the decision alternatives may be 
so large and, based on sensitivity analysis, appear so reliable, that a de- 
cision may be justified without further analysis. 

If analysis proceeds further, the risk estimations and risk reduction 
project data are fed into Task 6. 

Task 5 — Determine Range-of-Ef f ect . The prior tasks have been concerned 
with the assessment of the innovation over a limited range of application. 
Initially, it was the target applications identified by the innovator; in 
Task 4 a preliminary extended range-of -application was considered. Task 5 
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enab3.es the assessor to apply both the formal descriptors applied to indexing 
the innovation and the contextual knowledge about the innovation to the full 
range of the Navy's inventory of training courses, instruction vehicles, and 
job-tasks . 

ETAM stipulates that entities — training courses, job-tasks, instructional 
vehicles: — subject to range-of-ef f ect analysis may be indexed according to the 
rules of the descriptor taxonomy in ETAM formulated in Appendix A of this re- 
port. These indexed entities comprise a data base. This data base can be 
searched by search arguments composed of the descriptors that uniquely identify 
the relevant properties of the innovation. 

Thus, the indexed description of the innovation that was made in Task 1 
becomes an input into range-of -ef f ect search. When the search arguments reveal 
hits in the data base, the assessor examines contextual information about the 
entity (a course, a job-task, or an instructional vehicle) ,and makes a judgment 
of relevance or irrelevance. When matches between the innovation's properties 
and the entity's properties are made, the assessor estimates the kind and pro- 
portional magnitude of benefit/liability that is likely to be contributed by 
the innovation. A method from the initial ETAM study for describing a benefit 
(or liability) is shown in Appendix B (Figure B-5) . Ordinarily, there will be 
a number of benefit variab js and liability variables that make up a pattern 
or profile applicable to the innivation's promise. The identification of af- 
fected entities (courses, etc.) is input to cost-benefits analysis, Task 6. 

Task 6 — Perform Cost-Benefits Analysis . The decision tree developed in 
Task 4 is refined to include more precise costs and savings derived from pro- 
cessing the tangible benefits through the model. Thus, if the innovation has 
been estimated to enable an average reduction of 20% for learning the content 
of Course A to criterion, the cost model determines how frequently the course 
is taught, how many students take the course and, from its base of cost data 
about Course A, computes in dollars the actual projected savings. In addi- 
tion, the assessor uses utility scaling techniques for analyzing intangible 
benefits so that they are expressed in "equivalent dollars," thus enabling 
them to be combined into a single continuum of worth or value. Equivalent 
dollars is a utility expression rather than a literal dollar value. Proba- 
bilities of implementation success and user acceptance with and without the 
risk reduction projects are refined and the decision variables are 
recalculated. 

The procedural model described in the ETAM Phase II-B report, "Design of 
Trainiag Systems, The Development of Scaling Procedures," is the structure 
whereby the assessor generates multivariate utilities for outcomes in the 
decision tree . 

The model permits sensitivity analysis of variables that could reasonably 
change enough to affect the choice of a decision alternative. 

The output of this Task goes into Task 7, and subsequently to the de- 
cisionmaker in Task 8. 

Task 7 — Perform Financial Analysis . This task is concerned with assessing 
the tangible benefits and liabilities (those expressible in real dollars) in 
terms of certain economic measures. The investment costs and the annual costs 
and savings are calculated over a planning period extending a number of years 
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into the future. Rates-of-return on the invested dollars are determined for 
the incremental effect of each alternative compared tc the primary project. 
Alternatives consist of the proposed innovation, the existing system, and any 
other approaches defined in Task 2 which were considered reasonable candi- 
dates for further assessment. This assessment process provides a separate, 
distinct view of the value of the innovation from that gained from the de- 
cision tree assessment in Task 6. Both are inputs to the decisionmaker. 

The model enables sensitivity analysis. The purpose is to give the as- 
sessor insight into the variables that could cause a change in the decision 
if they were to vary over a reasonably expected range. 

Task 8— Hake the Accept/Reject/Study Decision . The immediate bases for 
the making of the decision are the financial analysis, plus sensitivity 
analysis, from Task 7, and the decision tree data, plus sensitivity analysis, 
from Task 6. 

However, the organized content of the data base files of the assessment 
project enables the decisionmaker to examine any of the constituent elements 
beneath the summary presentation made to him. He may substitute his own 
evaluations of worth, probability of outcomes, importance of intangibles, 
estimates of benefits or liabilities. He may M peel back" the data in each 
of the seven tasks by selectively accessing the files on each of these 
tasks. He could examine samples of range-of-ef f ect entities contained in 
computer files and retrieved interactively. 

The executive is thus in a position to put probes behind the facade of 
conclusions presented to him. He is therefore capable of reassurance in the 
results or direct participation in changing them according to his own values 
and store of information. 

Comment. The preceding description is merely a synoptic outline of the 
ETAM procedure. It is neither a substitute nor replacement for the full de- 
scription, including stipulations, assumptions, and caveats, that are con- 
tained in the source, the initial ETAM report, TAEG Report No. 12-3. 
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APPENDIX F 



WELSH'S DISSEMINATION METHODOLOGY 
Chapter III 

Rationale for the Purpose and Steps of the Methodology 



3 . 1 Introduction 

The purpose of this chapter is to provide a detailed justification for 
the purpose and each of the steps in the methodology. Literature from such 
sources as rural, medical, and general sociology? economics and marketing; 
and education will be cited in support of the steps included. 

Actually, the bulk of research reported in knowledge diffusion and uti- 
lization is concentrated in sociology and anthropology. Some work has been 
done in marketing research, although this is a much newer field. Relatively 
little dissemination research has been completed in education. The method- 
ology represents an attempt to synthesize the best available knowledge from 
the various disciplines mentioned. 

in the following pages, the purpose, and all the steps of Draft I of the 
dissemination methodology will be listed. This will be followed m turn by 
the justification of the purpose and the rationale for including each of the 
ten major steps. 

3.2 Dissemination Methodology: Draft I 

Purpose: To meet needs through the dissemination of products. 

Case I: The disseminator is working for a product developer (a special 
case— the disseminator is the product developer) 

Case II: The disseminator is working as an independent change agent 

(i.e., his/her remuneration would come from something like a 
university salary; dissemination is not his/her only major 
concern; rather, one of a number of interests) 

Case III: The disseminator is working for a funded agency whose function 
is to disseminate predicts (for example, the Far West Labora- 
tory for Educational Research & Development) 

Case IV: The disseminator is working for a consumer or group of con- 
sumers (e.g., a school system) 
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Negotiate a contract with a product developer interested in 
dissemination 

A. Explain each major step in the methodology to the product 
developer 

B. Identify the product to be disseminated 

C. Identify the resources available for the dissemination effort 

D. Prepare the contract and secure the product developer's final 
approval 

Plan the implementation of the remaining steps in the methodology 

Have the product developer design—or adapt, if the product is al- 
ready designed—the product to be as amenable to dissemination as 
possible, without changing the character of the product 

A. Determine the resources available for this step 

B. Make an initial judgment as to what general populations benefit 
from the adoption of the product 

C. Make the product as compatible with the potential adopter's 
values, culture, and/or traditions as possible 

1. Determine the values, culture, and/or traditions of the po- 
tential adopters 

2. Determine the adaptability of the product 

3. Adapt the product to the values, culture and jr traditions 
of the potential adopters 

D. Keep the cost of the product as low as possible 

1. If product costs nothing or almost nothing (e.g., a research 
report advocating some variety of behavior change) , move to 
Step III.E. 

2. Break the product down into component parts if possible 

3. Determine which of the components are essential to the prod- 
v.i:x. if it is to accomplish the purpose for which it was 
designed 

4. Eliminate those components found to be nonessential in 
Step 3 

5. Continue to break down the components until it is relatively 
easy to determine the lowest possible cost for each. The 
total will then be the lowest possible cost for the product 

6. Document cost information for use in Step V 



Reduce the complexity of the product as much as possible 

1. Steps III-D.2. through III.D.4. will have yielded compo- 
nents of the product. If the components are broken down 
as far as possible, go to Step 3 

2. Break down the components into their most basic subcomponents 

3. if necessary, provide explanation of the final list of com- 
ponents of the product 

4. Document complexity information for use in Step V 

Make the product "divisible," so that it can be tried initially 
on a small scale 

1. Determine whether the product is divisible or c; be made 
divisible without sacrificing its ability to ac .mplish its 
purpose. If it is not, or cannot be made divisible, go to 
Step III.G. 

2. Determine how the product can b.> tried on a limited basis 

a. Determine whether only part of the product need be tried 

b. Determine whether only a part of the adopting popula- 
tion (given that it is made up of more than one person) 
needs to try the product to give it a fair trial 

c. Document all possible ways the product can be made divis- 
ible for use in Step V 

Make the product observable, if possible, so that a potential 
adopter can see it in operation before he makes his decision 

1. Determine whether any institutions already use the product 

2. Determine whether the product developer or the disseminator 
can demonstrate the product 

' 3. Document observability for use in Step V 

Devise appropriate support services which the adopter may avail 
himself of after adoption of the product 

1. Determine potential difficulties adopters can encounter when 
using the product 

2. Determine which of these can be eliminated, or at least re- 
duced, by providing support services to the adopter 

3. Plan specifically support services to reduce problems identi- 
fied in Step 2 
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IV: Identify general populations that will benefit from the adoption of 
the product (potential adopters) ~ ** 

A. Determine the resources available for this step 

B. Identify general populations that have a need for the product 

1. Determine all populations that could possibly have a need 
for the product 

a. Read the relevant literature 

b. Talk with people whose work is in related areas 

c. Brainstorm all possible general populations 

2. Determine if the general populations identified in Step IV.B.l 
actually need the product 

a. Read relevant literature on these populations 

b. Talk with experts on these populations 

c. Sample opinions from the populations themselves 

d. Conduct relevant research on these populations 

3. Compile a list of populations that are identified as needing 
the product 

C. Among these populations, identify those sub-populations for whom 
the product fills a high-priority need 

1. Implement the needs analysis methodology, using at. least a 
sample of the target sub-population 

2. Determine whether or not the need the product fills has a 
sufficiently high priority on the needs of the population; 
if it does, go to Step IV. D. ; if not, select another sub- 
population and implement needs analysis again 

D. Of these, identify, as far as possible, those sub-populations 
on whom the product would have seriously detrimental side ef- 
fects, and leave them out of the dissemination effort 

1. If the resources are relatively small, make judgment from 
existing relevant knowledge 

a. Brainstorm possible side effects 

b. Talk to people knowledgeable about those sub-populations 

c. Read relevant literature on those sub-populations 

d. Sample opinions from the sub-populations 
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E. The above steps will result in a set of potential adopters who 
will be the target population; if it is different from the 
group identified in Step III.B., consider whether or not you 
need to recycle from Step III.C. on 

Identify, among the designated potential adopters, those subgroups 
most likely to react favorably to the product and focus communica- 
tion upon them 

A. Determine the resources available for this step 

B. Determine those in the population who are the early adopters 

1. Decide on definition of "early adopter" 

2. Identify products used by the target population similar to 
the product to be disseminated 

3. Determine those in target population who have a record of 
early adoption of those products 

a. Examine available records of adoption of those products 

b. Talk with those who use those products 

c. Talk with those connected with the adoption of those 
products 

4. Compile a list of those identified as "early adopters" 

C. If resources are relatively large, and if there are a relatively 
large number of early adopters, determine the opinion leaders 
among the early adopters. If not, go to Step V.D. 

1. Use other sociometric devices to identify opinion leaders 
(e.g., questionnaires that ask, "name the three colleagues 
from whom you would be most apt to seek advice with regard 
to (whatever the nature of the product is) u ) 

2. If the disseminator has insufficient expertise in interpret- 
ing sociomcsfcric devices (if sophisticated sociometric devices 
are in f ac- used) , employ an appropriate consultant 

3. Compile a final list of those members of the target popula- 
tion to be the first at whom dissemination efforts will be 
directed 

D. Develop a professional level (as opposed to friendship level) of 
rapport with the potential adopter identified in Step V.B.4. or 
Step V.C.3. 

1. Observe common rules of courtesy carefully (punctuality, 
politeness, etc.) 
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2. Remain honest and as objective as possible at all times 

3. Be aware of the potential adopter's professional activities, 
or the activities of his/her institution 

4. Make your interest (if genuine) in his/her activities or 
those of his/her institution known to the potential adopter 

5. Explain clearly to the potential adopter that your intent is 
to disseminate the product only to meet needs. If he/she 
does not see that it meets a need, you are not interested 

in disseminating the product to him/her 

6. Explain fully your role in disseminating the product 

7. Be able to explain readily any aspect of tie product 

E. Explain the product fully, and describe how it will meet the po- 
tential adopter's needs 

1. Explain your perception of the potential adopter's needs (or 
the needs of his/her system). If the potential adopter's 
diagnosis, and if the potential adopter and the disseminator 
cannot reach an agreement on needs, go to another potential 
adopter. Otherwise, proceed to Step 2 

2. Explain your perception of what the total impact of the prod- 
uct will be on the potential adopter's system 

a. Explain how you think it will meet need(s) 

b. Explain what negative effects may result 

3. Explain the characteristics of the product that were de- 
termined/developed in Step IV 

a. Explain the cost of the product 

b. Explain how the product can be observed in use (if it 
can) 

c. Explain how the product can be tried on a limited basis 
(if it can) 

d. Explain its compatibility with the cultures, values, and 
traditions of the potential adopter (if it is, in fact, 
compatible) 

e. Explain the support services available for use if the 
product is adopted 

If the potential adopter (s) decide (s) to adopt, make the product 
available to him/her as soon as possible, including all available 
support services if they are desired 
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VII: If resources for this step remain, implement the "2-step model" 
i.e., help the opinion leaders disseminate the product to others 
in the population 

A. Determine whether the opinion leader wants to help in the dis- 
semination effort 

B. Determine whether the opinion leader is to be trusted with the 
resources available for this step. If not, go to Step VIII 

r* 

C. Determine how much and what kinds of resources the opinion 
leader needs 

D. Make the resources available to the opinion leader 
VIII: Evaluate the results of the adoption/rejection 

A. The Fortune-Hutchinson evaluation methodology is recommended 
with the product developer as the decisionmaker 

B. If adopted, evaluate its acceptance, use, and impact, including 
unintended outcomes 

1. If it meets the adopter's need, proceed with other potential 
adopters in the same manner — i.e., return to Step IV 

2. If it does not meet the need, or for some other reason causes 
trouble for the adopter, return to Step III 

C. If rejected, evaluate reason (s) for rejection and return to 
Step III or IV, as the product developer decides (i.e., he may 
choose either to redesign his product or to aim the existing 
product at a different target population) 

IX: Proceed through Steps IV VIII until the product is complete ly dissemi- 
nated, or until resources run out 

X: Evaluate the success of the methodology and revise where appropriate 
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APPENDIX G 



SUPPLEMENTAL INFORMATION NEEDED FOR EVALUATION 



To broaden the scope of the evaluation of technology transfer in the Army , 
supplemental information should be obtained. The types of information needed 
are suggested below: 

1. Within ARI , one could evaluate research products in general (of 
which training products form a subset) . An initial step here is to categor- 
ize those products with which ARI is involved. For example, one could sub- 
jectively "factor analyze" the research products reported by Drucker (1977) 
into four major areas, based on their purpose (see Table G.l). Then, within 
each category, one could focus interest on completed products and those in 
the process of being completed (i.e., handed off from 6.3 to 6.4 budgetary 
and operational control). Within each of the completed products, determine 
successful (S) versus nonsuccessf ul (N) products. Success can be determined 
by objective sources (e.g., training effectiveness data, usage in terms of 
frequency counts) and by subjective sources (e.g., obtaining statements from 
different levels of users). Within the process products, select products 
that have a very high probability of being successful (+) and those that 
may or may not be successful (?) . Finally, track (rate) all the products 

in terms of the organizational events based on Yin et al. (1978) approach 
(e.g., incorporation and maintenance of products over time). The completed 
products' outcomes will be predictive markers for the process products 1 out- 
comes. That is, if the completed S and TSI products cluster discriminate^ on 
Yin's organizational events, then this institutionalization approach may be 
used to predict the outcome of the process + and ? products. The relevance 
of this approach is that it may help to highlight the processes that occur 
during the handoff of ARI products to the user from 6.3 to 6.4 RDT&E funding 
and operational control. If the "6. 3-6.4" interval is critical for technology 
transfer, then perhaps application of Yin's "routinization" approach may help 
to correlate organizational events with S and N products. This proposed analy- 
sis is schematized in Table G.2. 

It is realized thau. this proposed analysis takes ARI out of its formal 
RDT&E boundary (past 6.3), but if ARI focuses only on the "6.3 handoff" (user 
acceptance) while still being held accountable for its institutionalization 
(6.4-6.7), events influencing this accountability (i.e., effecting the out- 
come of the S and N of products) may be unnoticed during the 6.3-6.4 interval. 
A major implication of previous assignment of accountability to the HR RDT&E 
community (see Note 2) is that the handling of research products through the 
6.4 to 6.7 phases does not cause significant transformations on the eventual 
outcomes associated with the use of the product. Therefore, serious consid- 
eration should be given to studying those events that not only correlate with 
later incorporation (6.4 to 6.7), but also those during the 6.3-6.4 interval 
(i.e., the utilization stage of the institutionalization activity). 

2. Supplemental to the first suggestion would be information depicting 
those Army/DOD agencies that use (e.g., spend) RDT&E funds, and the types of 
RDT&E funds used. This suggestions is a matrix consisting of 6.1 through 6.7 
funds listed along one dimension, and the names of the agencies along the 
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Table G.l 

Proposed Categories of ARI Research Products 



1. Informative-Explanatory ("What is") 
Data banks 

Job aid, information 

Making the researcher an expert 

New technological information v 
Project report 

Report suitable for publication in professional journal (Scientific 
v Report) 

Research findings (data) 

Research material for university-level education 

2. Informative-Procedural ("How to n ) 

Guidelines 
Handbook 
Policy ma.mal 

SOP (Steading operating procedures) 
Technical manual 

Training literature (manuals, circulars, instructions) 

3. Recommendations (this is a subset of both 1 and 2 above, depending on its 

purpose) 

Recommendations for change in doctrine 
Recommendations for change in policy 

4. Research Methodology 
Research tool 

Tests and measurement instruments 

5. Training Products 

Training literature (also listed under 2) • 
Training device 
Training program 



Source: Based on Drucker , 1977, 
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Table G.2 

Tracking the Outcome of Selected Research Products 



Research products 



History 



Outcome 



Completed 



Success 



Nonsuccess 



Informative-explanatory 



Process 



Completed 



N 



Informative-procedural 



Process 



Completed 



Research methodology 



Process 



Completed 



N 



Training product 



Process 



Note: Document (rate) products on the organizational events during institu- 
tionalization as listed by Yin et al. (1978) . 
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other dimension. One could look at agencies involved in HR RDT&E research 
products in general, or one could delimit the field of interest to training 
products. The question of interest is f rom-whom-to-whom are funds handed 
off during the transition of RDT&E funding control of the research product. 
This information would indicate how much interaction one should have with 
specific agencies who are involved in the handoff of budgetary and operational 
control of the research product. For example, one could see how many agencies 
spend/use funds during the 6.3-6.4 interval. This matrix would serve as a 
"funding flow" chart consisting of clusters of agencies which may be corre- 
lated with organizational events critical to the outcome of the research 
product. 



3. An a priori baseline of success rates of research product utilization 
would help to guide subsequent funding and research directions. Determining 
the baseline rate has not been addressed empirically. First, there is a defi- 
nition issue. This could be resolved by using both objective and subjective 
sources of information, as mentioned previously. Then the scope must be ad- 
dressed. The scope could be limited up to 6.3 (user acceptance), or could in- 
clude the transition period (6.3-6.4 interval) of budgetary and operational 
control. Also, a within-Army standard could be referenced from the "hardware" 
RDT&E community (e.g., the Harry Diamond Labs), using their "success" rates 
for comparison wit.'i those of "software" RDT&E produces. 

It is suggested that the above-mentioned supplemental information, as 
well as the proposed evaluation scheme, may be incorporated with the product 
utilization activity of the Plans, Programs, and Operations (PPO) office of 
ARI. The above-mentioned suggestions might provide a conceptual framework 
from which the PPO office could decide future evaluative efforts on research 
product utilization. ^ 
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APPENDIX H 
FORMAT FOR AN IMPLEMENTATION PLAN* 



2. 



3. 



4. 



5. 



irements Analysis 

General Nature of the Problem 

a. Description of current training problem 

b. What operational problems exist 

Reasons for Attention at This Time 

a. Congressional concern 

b. Budget?ry reasons 

c. Military readiness 

d. As appropriate 

Objectives Related to the Problem 

a. Use established statement of objectives, or, where necessary, a ver- 
sion of it specially tailored to the operational problem. 

b. Three essential components 

(1) Good effect or result intended to be achieved 

(2) For whom 

(3) By what means 

Measure of Effectiveness (MOE's) Related to the Objective 

a. If the plan relates to a particular training program, use the estab- 
?.ished MOE's supplemented as necessary by others more directly related 
to the hypothesis being tested. 

b. Attempt to use quantitative indicators of objectives results. 

c. Don't confuse MOE's with activity levels. 
Target Groups - Receiver Level of User 

a. Present level of training effectiveness 

b. Estimated potential level of training effectiveness 

Beneficiary Group - Any/All Level (s) of User 



*Extracted and modified from Keller (1979) . 
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7. Programs Related to or Affected by this Problem 

a. Local agency 

b. Command 

c. As appropriate 

8. Alternative Considered 



Dissemination and Institutionalization 

9. How the Approved Alternative Will Operate (the detailed operational 
plan) 

a. Staffing 

b. Organization 

c. Relations with other organizations 

d. Policies 

e. Method of operating 

(1) Chronological account of a typical operation 

(2) Special techniques used 

10. Estimated Effectiveness of the Approved Alternative 

a. Refer back to the MOE 1 s in 4, above. 

b. Refer back to the discussion in the General Nature of the Problem 
in 1 , above . 

11. Estimated Costs of the Approved Alternative 

a. Investment 

(1) R&D 

(2) Facilities 

(3) Equipment 

(4) Initial training 

b. Operating 

(1) Personnel 

(2) Minor equipment 

(3) Supplies 

(4) Direct services 

(5) Contracted services 

c . Totals — Annual 
Totals — 10 year 

(1) Indicate whether the costs are in constant or current dollars 

(2) Indicate whether present value discounting has been done. 



As applicable 
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12. 



Spillover Effects 



a. Refer to the public agencies listed in 7, above. 

b. Other spillovers 

c. Coordination problems between levels of users 
13. Other Considerations 

a. Resistance 

b. Constraints on operations V 

c. Linkage agent role for sponsor's representative 

d. Training 

e. Continued fiscal backing 



RDT&E Concerns 

3 4. Proposed Method of Evaluating the Approved Alternative 

a. In drawing up an evaluation scheme (or research design) , one must 
allow for certain basic concerns, to make judgments subsequently 
about the overall validity of the evaluation. These include 

•1) The nature of the questions to be answered or the decisions to 
to be made. 

(2) What will constitute a "success" or a "failure." 

(3) The basic methodology to be employed, including consideration 
of the bases of comparison and the adequacy and power of the 
specific techniques to be used. 

(4) The level of confidence which can be attached to the conclusions. 

(5) The generalizability of the conclusions. 

b. In dealing with these basic concerns the following more specific 
procedures and questions should be dealt with in full detail: 

(1) What are the hypotheses to be tested? 

(a) How do they relate to the stated objectives? 

(b) What are the logical linkages between the overall good ef- 
fect and the immediately testable good effects? 

(c) Are there assumptions which can or should be converted to 
testable hypotheses? 

(2) What measures of effectiveness are related to the hypotheses to 
be tested? 

(a) Are these direct measures of effectiveness or are they 
proxy measures? 
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(b) If they are proxy measures, what is the basis in evidence 
for the presumed relat ionship betwee.: the proxy measure 
and the direct measure? 

Data on what other measurable consequences and items of evalua- 
tive interest need to be collected? 

How frequently are data on the measures of e f fectiveness 9 other 
measurable consequences, and other items of evaluative interest 
to be gathered? 

For how long will each category of data be collected? 
How, exactly, will all of these data be collected? 

(a) By whom; and will they need any special training in data 
collection procedures? What opportunities for bias are 
there amor:g the data collectors? 

(b) What procedures will be followed in collecting the data? 
Do they create the possibility of bias creeping in? 

(c) What forms will be used to collect the data? Are they 
clear enough? Will Privacy Act requirements inhibit the 
data collection effort? 

With respect to the experimental group: 

(a) What criteria will be used to select subjects intc the 
experimental group? 

(b) How will the sample be stratified? 

(c) What will be the maximum and minimum usable sample sizes? 

(d) What methods or mechanisms will be used for actually se- 
lecting subjects in accordance with the criteria specified 
in a, b, and c, above? 

(e) To what degree can the Hawthorne Effect be eliminated? 

With respect to the control group used as a basis of comparison 
with the experimental group: 

(a) What exactly will be the form of comparison? 

(b) How will points 7a, b, c, and d, above, be dealt with for 
the control group? 

(c) Are there ethical or legal problems involved in choosing or 
using the control group? 
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(9) Exactly what calculations will be made using the data to be 
gathered? 

(a) Are the statistical or other techniques to be used appro- 
priate to the problem? 

(b) Are they sufficiently powerful to produce the desired 
results? 

(c) Have all of the data required to drive the computational 
models been identified? 

(d) Have provisions been made to assure the availability of 
all of these data? 

(e) If some of the data cannot be collected at all, or with 
sufficient reliability, what fall-back computational pro- 
cedures can be used? 

(10) Meeting what criteria, or ranges of values which result from 

the calculations, will be considered as constituting a "success"? 
an "indeterminate result"? a "failure"? 
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